[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786334794676589-31872-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 04:06:34 dist-test-slave-8w9v krb5kdc[31881](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 04:06:34 dist-test-slave-8w9v krb5kdc[31881](info): set up 2 sockets
Aug 10 04:06:34 dist-test-slave-8w9v krb5kdc[31881](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260810 04:06:35.893514 31872 mini_kdc.cc:121] Time spent starting KDC: real 1.187s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786334794676589-31872-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786334794676589-31872-0/krb5kdc/test-user.keytab.
Aug 10 04:06:35 dist-test-slave-8w9v krb5kdc[31881](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334795, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T04:06:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T04:06:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.32.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.32.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.32.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.62.keytab.
Entry for principal kudu/127.31.32.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.62.keytab.
I20260810 04:06:36.023170 31872 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskws89Io/build/release/bin/kudu
/tmp/dist-test-taskws89Io/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786334794676589-31872-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786334794676589-31872-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786334794676589-31872-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786334794676589-31872-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.32.62:41015
--webserver_interface=127.31.32.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.62.keytab
--principal=kudu/127.31.32.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.32.20:38041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.32.62:41015
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786334794676589-31872-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786334794676589-31872-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786334794676589-31872-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:06:36.117336 31894 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:06:36.117555 31894 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:06:36.117594 31894 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:06:36.118767 31894 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:06:36.118832 31894 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:06:36.118860 31894 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:06:36.118881 31894 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:06:36.118913 31894 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:06:36.120653 31894 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.32.20:38041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786334794676589-31872-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786334794676589-31872-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.32.62:41015
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.62.keytab
--principal=kudu/127.31.32.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.32.62:41015
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786334794676589-31872-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.32.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31894
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786334794676589-31872-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:06:36.121070 31894 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:06:36.121338 31894 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:06:36.127269 31903 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:06:36.127324 31901 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:06:36.127453 31894 server_base.cc:1034] running on GCE node
W20260810 04:06:36.127975 31900 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:06:36.128352 31894 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:06:36.128738 31894 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:06:36.129908 31894 hybrid_clock.cc:648] HybridClock initialized: now 1786334796129887 us; error 77 us; skew 500 ppm
Aug 10 04:06:36 dist-test-slave-8w9v krb5kdc[31881](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334796, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:06:36.132619 31894 init.cc:377] Logged in from keytab as kudu/127.31.32.62@KRBTEST.COM (short username kudu)
I20260810 04:06:36.133209 31894 webserver.cc:460] Webserver started at http://127.31.32.62:43375/ using document root <none> and password file <none>
I20260810 04:06:36.133563 31894 fs_manager.cc:359] Metadata directory not provided
I20260810 04:06:36.133641 31894 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:06:36.133847 31894 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:06:36.134977 31894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786334794676589-31872-0/minicluster-data/master-0/data/instance:
uuid: "abaab99011c041868056df33c3abc533"
format_stamp: "Formatted at 2026-08-10 04:06:36 on dist-test-slave-8w9v"
server_key: "131efdfc7e2c38c8aca8476337fe2ea0"
server_key_iv: "5aa4ca25f289bb68754f5e1c2b27a377"
server_key_version: "encryptionkey@0"
I20260810 04:06:36.135334 31894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786334794676589-31872-0/minicluster-data/master-0/wal/instance:
uuid: "abaab99011c041868056df33c3abc533"
format_stamp: "Formatted at 2026-08-10 04:06:36 on dist-test-slave-8w9v"
server_key: "131efdfc7e2c38c8aca8476337fe2ea0"
server_key_iv: "5aa4ca25f289bb68754f5e1c2b27a377"
server_key_version: "encryptionkey@0"
I20260810 04:06:36.137180 31894 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260810 04:06:36.138185 31910 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:06:36.138454 31894 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:06:36.138566 31894 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786334794676589-31872-0/minicluster-data/master-0/data,/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786334794676589-31872-0/minicluster-data/master-0/wal
uuid: "abaab99011c041868056df33c3abc533"
format_stamp: "Formatted at 2026-08-10 04:06:36 on dist-test-slave-8w9v"
server_key: "131efdfc7e2c38c8aca8476337fe2ea0"
server_key_iv: "5aa4ca25f289bb68754f5e1c2b27a377"
server_key_version: "encryptionkey@0"
I20260810 04:06:36.138644 31894 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786334794676589-31872-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786334794676589-31872-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786334794676589-31872-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:06:36.159459 31894 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:06:36.161655 31894 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:06:36.161864 31894 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:06:36.166638 31894 rpc_server.cc:307] RPC server started. Bound to: 127.31.32.62:41015
I20260810 04:06:36.166738 31964 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.32.62:41015 every 8 connection(s)
I20260810 04:06:36.167330 31894 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786334794676589-31872-0/minicluster-data/master-0/data/info.pb
I20260810 04:06:36.168758 31965 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:06:36.169852 31872 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskws89Io/build/release/bin/kudu as pid 31894
I20260810 04:06:36.169965 31872 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786334794676589-31872-0/minicluster-data/master-0/wal/instance
I20260810 04:06:36.170563 31872 external_mini_cluster.cc:1428] Setting key 3934d7d6540612e286826d491dd4048a
I20260810 04:06:36.172260 31965 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P abaab99011c041868056df33c3abc533: Bootstrap starting.
I20260810 04:06:36.173204 31965 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P abaab99011c041868056df33c3abc533: Neither blocks nor log segments found. Creating new log.
I20260810 04:06:36.173561 31965 log.cc:824] T 00000000000000000000000000000000 P abaab99011c041868056df33c3abc533: Log is configured to *not* fsync() on all Append() calls
I20260810 04:06:36.174396 31965 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P abaab99011c041868056df33c3abc533: No bootstrap required, opened a new log
I20260810 04:06:36.177502 31965 raft_consensus.cc:348] T 00000000000000000000000000000000 P abaab99011c041868056df33c3abc533 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abaab99011c041868056df33c3abc533" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 41015 } }
I20260810 04:06:36.177659 31965 raft_consensus.cc:374] T 00000000000000000000000000000000 P abaab99011c041868056df33c3abc533 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:06:36.177740 31965 raft_consensus.cc:729] T 00000000000000000000000000000000 P abaab99011c041868056df33c3abc533 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abaab99011c041868056df33c3abc533, State: Initialized, Role: FOLLOWER
I20260810 04:06:36.177925 31965 consensus_queue.cc:260] T 00000000000000000000000000000000 P abaab99011c041868056df33c3abc533 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abaab99011c041868056df33c3abc533" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 41015 } }
I20260810 04:06:36.178059 31965 raft_consensus.cc:388] T 00000000000000000000000000000000 P abaab99011c041868056df33c3abc533 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 04:06:36.178107 31965 raft_consensus.cc:482] T 00000000000000000000000000000000 P abaab99011c041868056df33c3abc533 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 04:06:36.178159 31965 raft_consensus.cc:3037] T 00000000000000000000000000000000 P abaab99011c041868056df33c3abc533 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:06:36.178884 31965 raft_consensus.cc:504] T 00000000000000000000000000000000 P abaab99011c041868056df33c3abc533 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abaab99011c041868056df33c3abc533" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 41015 } }
I20260810 04:06:36.179149 31965 leader_election.cc:302] T 00000000000000000000000000000000 P abaab99011c041868056df33c3abc533 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: abaab99011c041868056df33c3abc533; no voters:
I20260810 04:06:36.179395 31965 leader_election.cc:288] T 00000000000000000000000000000000 P abaab99011c041868056df33c3abc533 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 04:06:36.179458 31971 raft_consensus.cc:2781] T 00000000000000000000000000000000 P abaab99011c041868056df33c3abc533 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:06:36.179611 31971 raft_consensus.cc:686] T 00000000000000000000000000000000 P abaab99011c041868056df33c3abc533 [term 1 LEADER]: Becoming Leader. State: Replica: abaab99011c041868056df33c3abc533, State: Running, Role: LEADER
I20260810 04:06:36.179768 31971 consensus_queue.cc:237] T 00000000000000000000000000000000 P abaab99011c041868056df33c3abc533 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abaab99011c041868056df33c3abc533" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 41015 } }
I20260810 04:06:36.179869 31965 sys_catalog.cc:564] T 00000000000000000000000000000000 P abaab99011c041868056df33c3abc533 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:06:36.180315 31973 sys_catalog.cc:455] T 00000000000000000000000000000000 P abaab99011c041868056df33c3abc533 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "abaab99011c041868056df33c3abc533" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abaab99011c041868056df33c3abc533" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 41015 } } }
I20260810 04:06:36.180421 31973 sys_catalog.cc:458] T 00000000000000000000000000000000 P abaab99011c041868056df33c3abc533 [sys.catalog]: This master's current role is: LEADER
I20260810 04:06:36.180815 31974 sys_catalog.cc:455] T 00000000000000000000000000000000 P abaab99011c041868056df33c3abc533 [sys.catalog]: SysCatalogTable state changed. Reason: New leader abaab99011c041868056df33c3abc533. Latest consensus state: current_term: 1 leader_uuid: "abaab99011c041868056df33c3abc533" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abaab99011c041868056df33c3abc533" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 41015 } } }
I20260810 04:06:36.180965 31974 sys_catalog.cc:458] T 00000000000000000000000000000000 P abaab99011c041868056df33c3abc533 [sys.catalog]: This master's current role is: LEADER
I20260810 04:06:36.181334 31979 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 04:06:36.182615 31979 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 04:06:36.185041 31979 catalog_manager.cc:1347] Generated new cluster ID: 73a83f66f8054a02b9c267cdac5fac4c
I20260810 04:06:36.185092 31979 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 10 04:06:36 dist-test-slave-8w9v krb5kdc[31881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334795, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.32.62@KRBTEST.COM
I20260810 04:06:36.192641 31967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:36.171737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53900 (local address 127.31.32.62:41015)
0810 04:06:36.172068 (+ 331us) server_negotiation.cc:206] Beginning negotiation
0810 04:06:36.172078 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 04:06:36.172282 (+ 204us) server_negotiation.cc:407] Connection header received
0810 04:06:36.174514 (+ 2232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:06:36.174521 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:06:36.174609 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:06:36.174726 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:06:36.177137 (+ 2411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:36.178341 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:36.180063 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:36.180716 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:36.180955 (+ 239us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:06:36.187578 (+ 6623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:06:36.187582 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:06:36.187582 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:06:36.187594 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:06:36.190024 (+ 2430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:06:36.190531 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:06:36.190534 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:06:36.190534 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:06:36.190643 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:06:36.190937 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:06:36.190938 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:06:36.190938 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:06:36.191243 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:06:36.191526 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0810 04:06:36.192059 (+ 533us) server_negotiation.cc:299] Negotiation successful
0810 04:06:36.192270 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":80,"threads_started":1}
I20260810 04:06:36.205477 31979 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 04:06:36.206081 31979 catalog_manager.cc:1504] Loading token signing keys...
I20260810 04:06:36.210907 31979 catalog_manager.cc:5868] T 00000000000000000000000000000000 P abaab99011c041868056df33c3abc533: Generated new TSK 0
I20260810 04:06:36.211102 31979 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.32.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.32.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.32.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.1.keytab.
Entry for principal kudu/127.31.32.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.1.keytab.
I20260810 04:06:36.243817 31872 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskws89Io/build/release/bin/kudu
/tmp/dist-test-taskws89Io/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786334794676589-31872-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786334794676589-31872-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786334794676589-31872-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786334794676589-31872-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.32.1:0
--local_ip_for_outbound_sockets=127.31.32.1
--webserver_interface=127.31.32.1
--webserver_port=0
--tserver_master_addrs=127.31.32.62:41015
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.1.keytab
--principal=kudu/127.31.32.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.32.20:38041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786334794676589-31872-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786334794676589-31872-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786334794676589-31872-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:06:36.367506 31996 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:06:36.367697 31996 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:06:36.367720 31996 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:06:36.368934 31996 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:06:36.368978 31996 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:06:36.369005 31996 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.32.1
I20260810 04:06:36.370529 31996 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.32.20:38041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786334794676589-31872-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786334794676589-31872-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.1.keytab
--principal=kudu/127.31.32.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.32.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786334794676589-31872-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.32.1
--webserver_port=0
--tserver_master_addrs=127.31.32.62:41015
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31996
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.32.1
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786334794676589-31872-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:06:36.370857 31996 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:06:36.371130 31996 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:06:36.377583 32002 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:06:36.377645 32003 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:06:36.377900 32005 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:06:36.378016 31996 server_base.cc:1034] running on GCE node
I20260810 04:06:36.378224 31996 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:06:36.378501 31996 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:06:36.379674 31996 hybrid_clock.cc:648] HybridClock initialized: now 1786334796379653 us; error 36 us; skew 500 ppm
Aug 10 04:06:36 dist-test-slave-8w9v krb5kdc[31881](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334796, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:06:36.382937 31996 init.cc:377] Logged in from keytab as kudu/127.31.32.1@KRBTEST.COM (short username kudu)
I20260810 04:06:36.383412 31996 webserver.cc:460] Webserver started at http://127.31.32.1:37851/ using document root <none> and password file <none>
I20260810 04:06:36.383665 31996 fs_manager.cc:359] Metadata directory not provided
I20260810 04:06:36.383733 31996 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:06:36.383905 31996 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:06:36.385100 31996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786334794676589-31872-0/minicluster-data/ts-0/data/instance:
uuid: "cda01427493e4d22845c3dbad5a1620b"
format_stamp: "Formatted at 2026-08-10 04:06:36 on dist-test-slave-8w9v"
server_key: "b87bce82b24a7c1da0fe418d5ae474d3"
server_key_iv: "82adbe8126b0cf495131847ed33b2e00"
server_key_version: "encryptionkey@0"
I20260810 04:06:36.385509 31996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786334794676589-31872-0/minicluster-data/ts-0/wal/instance:
uuid: "cda01427493e4d22845c3dbad5a1620b"
format_stamp: "Formatted at 2026-08-10 04:06:36 on dist-test-slave-8w9v"
server_key: "b87bce82b24a7c1da0fe418d5ae474d3"
server_key_iv: "82adbe8126b0cf495131847ed33b2e00"
server_key_version: "encryptionkey@0"
I20260810 04:06:36.386986 31996 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 04:06:36.387820 32014 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:06:36.388021 31996 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:06:36.388149 31996 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786334794676589-31872-0/minicluster-data/ts-0/data,/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786334794676589-31872-0/minicluster-data/ts-0/wal
uuid: "cda01427493e4d22845c3dbad5a1620b"
format_stamp: "Formatted at 2026-08-10 04:06:36 on dist-test-slave-8w9v"
server_key: "b87bce82b24a7c1da0fe418d5ae474d3"
server_key_iv: "82adbe8126b0cf495131847ed33b2e00"
server_key_version: "encryptionkey@0"
I20260810 04:06:36.388232 31996 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786334794676589-31872-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786334794676589-31872-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786334794676589-31872-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:06:36.397150 31996 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:06:36.399135 31996 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:06:36.399295 31996 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:06:36.399574 31996 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:06:36.399889 31996 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:06:36.399996 31996 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:06:36.400049 31996 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:06:36.400074 31996 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:06:36.406607 31996 rpc_server.cc:307] RPC server started. Bound to: 127.31.32.1:44177
I20260810 04:06:36.406692 32137 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.32.1:44177 every 8 connection(s)
I20260810 04:06:36.407039 31996 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786334794676589-31872-0/minicluster-data/ts-0/data/info.pb
I20260810 04:06:36.410209 31872 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskws89Io/build/release/bin/kudu as pid 31996
I20260810 04:06:36.410288 31872 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786334794676589-31872-0/minicluster-data/ts-0/wal/instance
I20260810 04:06:36.410457 31872 external_mini_cluster.cc:1428] Setting key 9251e4a8986056378ad46ba770ce5ef9
Aug 10 04:06:36 dist-test-slave-8w9v krb5kdc[31881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334796, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.1@KRBTEST.COM for kudu/127.31.32.62@KRBTEST.COM
I20260810 04:06:36.418133 32141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:36.407930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:41015 (local address 127.31.32.1:55509)
0810 04:06:36.408138 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0810 04:06:36.408178 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0810 04:06:36.408734 (+ 556us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:06:36.408991 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:06:36.408995 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:06:36.409774 (+ 779us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:06:36.409975 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:06:36.409980 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:06:36.411411 (+ 1431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:06:36.411412 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:06:36.412305 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:06:36.412308 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:06:36.412856 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:06:36.412857 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:06:36.412967 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:06:36.413538 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:06:36.413546 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:06:36.414917 (+ 1371us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:06:36.416835 (+ 1918us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:06:36.416841 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:06:36.416842 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:06:36.417056 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:06:36.417276 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:06:36.417276 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:06:36.417276 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:06:36.417374 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:06:36.417654 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:06:36.417659 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:06:36.417769 (+ 110us) client_negotiation.cc:769] Sending connection context
0810 04:06:36.417860 (+ 91us) client_negotiation.cc:240] Negotiation successful
0810 04:06:36.417976 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":93,"threads_started":1}
I20260810 04:06:36.418124 31967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:36.407964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.1:55509 (local address 127.31.32.62:41015)
0810 04:06:36.408054 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0810 04:06:36.408056 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:06:36.408263 (+ 207us) server_negotiation.cc:407] Connection header received
0810 04:06:36.408850 (+ 587us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:06:36.408851 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:06:36.408863 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:06:36.408920 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:06:36.410652 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:36.411310 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:36.412391 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:36.412738 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:36.412830 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:06:36.415025 (+ 2195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:06:36.415026 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:06:36.415026 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:06:36.415034 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:06:36.416731 (+ 1697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:06:36.417133 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:06:36.417134 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:06:36.417134 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:06:36.417202 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:06:36.417439 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:06:36.417440 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:06:36.417441 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:06:36.417586 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:06:36.417661 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0810 04:06:36.417908 (+ 247us) server_negotiation.cc:299] Negotiation successful
0810 04:06:36.418001 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 04:06:36.418731 32139 heartbeater.cc:347] Connected to a master server at 127.31.32.62:41015
I20260810 04:06:36.418861 32139 heartbeater.cc:464] Registering TS with master...
I20260810 04:06:36.419045 32139 heartbeater.cc:511] Master 127.31.32.62:41015 requested a full tablet report, sending...
I20260810 04:06:36.420065 31927 ts_manager.cc:194] Registered new tserver with Master: cda01427493e4d22845c3dbad5a1620b (127.31.32.1:44177)
WARNING: no policy specified for kudu/127.31.32.2@KRBTEST.COM; defaulting to no policy
I20260810 04:06:36.421005 31927 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.32.1@KRBTEST.COM'} at 127.31.32.1:55509
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.32.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.32.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.2.keytab.
Entry for principal kudu/127.31.32.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.2.keytab.
I20260810 04:06:36.438745 31872 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskws89Io/build/release/bin/kudu
/tmp/dist-test-taskws89Io/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786334794676589-31872-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786334794676589-31872-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786334794676589-31872-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786334794676589-31872-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.32.2:0
--local_ip_for_outbound_sockets=127.31.32.2
--webserver_interface=127.31.32.2
--webserver_port=0
--tserver_master_addrs=127.31.32.62:41015
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.2.keytab
--principal=kudu/127.31.32.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.32.20:38041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786334794676589-31872-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786334794676589-31872-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786334794676589-31872-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:06:36.522711 32144 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:06:36.522897 32144 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:06:36.522939 32144 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:06:36.524076 32144 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:06:36.524152 32144 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:06:36.524217 32144 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.32.2
I20260810 04:06:36.525691 32144 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.32.20:38041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786334794676589-31872-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786334794676589-31872-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.2.keytab
--principal=kudu/127.31.32.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.32.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786334794676589-31872-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.32.2
--webserver_port=0
--tserver_master_addrs=127.31.32.62:41015
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32144
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.32.2
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786334794676589-31872-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:06:36.525924 32144 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:06:36.526153 32144 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:06:36.531289 32154 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:06:36.531301 32150 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:06:36.531289 32151 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:06:36.531306 32144 server_base.cc:1034] running on GCE node
I20260810 04:06:36.531888 32144 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:06:36.532194 32144 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:06:36.533450 32144 hybrid_clock.cc:648] HybridClock initialized: now 1786334796533439 us; error 31 us; skew 500 ppm
Aug 10 04:06:36 dist-test-slave-8w9v krb5kdc[31881](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334796, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:06:36.536222 32144 init.cc:377] Logged in from keytab as kudu/127.31.32.2@KRBTEST.COM (short username kudu)
I20260810 04:06:36.536696 32144 webserver.cc:460] Webserver started at http://127.31.32.2:45345/ using document root <none> and password file <none>
I20260810 04:06:36.536952 32144 fs_manager.cc:359] Metadata directory not provided
I20260810 04:06:36.537029 32144 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:06:36.537187 32144 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:06:36.538199 32144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786334794676589-31872-0/minicluster-data/ts-1/data/instance:
uuid: "3bf4bbc3d28b40ecb89251365a12d969"
format_stamp: "Formatted at 2026-08-10 04:06:36 on dist-test-slave-8w9v"
server_key: "e4df0dc93b2fa8f264028fe6528648e3"
server_key_iv: "b3748ad3aab75bf37aaa064e05c1d395"
server_key_version: "encryptionkey@0"
I20260810 04:06:36.538571 32144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786334794676589-31872-0/minicluster-data/ts-1/wal/instance:
uuid: "3bf4bbc3d28b40ecb89251365a12d969"
format_stamp: "Formatted at 2026-08-10 04:06:36 on dist-test-slave-8w9v"
server_key: "e4df0dc93b2fa8f264028fe6528648e3"
server_key_iv: "b3748ad3aab75bf37aaa064e05c1d395"
server_key_version: "encryptionkey@0"
I20260810 04:06:36.539990 32144 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260810 04:06:36.540866 32161 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:06:36.541065 32144 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 04:06:36.541149 32144 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786334794676589-31872-0/minicluster-data/ts-1/data,/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786334794676589-31872-0/minicluster-data/ts-1/wal
uuid: "3bf4bbc3d28b40ecb89251365a12d969"
format_stamp: "Formatted at 2026-08-10 04:06:36 on dist-test-slave-8w9v"
server_key: "e4df0dc93b2fa8f264028fe6528648e3"
server_key_iv: "b3748ad3aab75bf37aaa064e05c1d395"
server_key_version: "encryptionkey@0"
I20260810 04:06:36.541201 32144 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786334794676589-31872-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786334794676589-31872-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786334794676589-31872-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:06:36.552563 32144 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:06:36.554272 32144 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:06:36.554381 32144 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:06:36.554548 32144 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:06:36.554782 32144 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:06:36.554811 32144 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:06:36.554832 32144 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:06:36.554845 32144 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:06:36.561268 32144 rpc_server.cc:307] RPC server started. Bound to: 127.31.32.2:33771
I20260810 04:06:36.561317 32281 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.32.2:33771 every 8 connection(s)
I20260810 04:06:36.561607 32144 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786334794676589-31872-0/minicluster-data/ts-1/data/info.pb
I20260810 04:06:36.563326 31872 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskws89Io/build/release/bin/kudu as pid 32144
I20260810 04:06:36.563396 31872 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786334794676589-31872-0/minicluster-data/ts-1/wal/instance
I20260810 04:06:36.563570 31872 external_mini_cluster.cc:1428] Setting key cef527e3110582d84e28a5cc78ac62c9
Aug 10 04:06:36 dist-test-slave-8w9v krb5kdc[31881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334796, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.2@KRBTEST.COM for kudu/127.31.32.62@KRBTEST.COM
I20260810 04:06:36.572041 31967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:36.562304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.2:58181 (local address 127.31.32.62:41015)
0810 04:06:36.562390 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0810 04:06:36.562392 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:06:36.562560 (+ 168us) server_negotiation.cc:407] Connection header received
0810 04:06:36.563027 (+ 467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:06:36.563028 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:06:36.563038 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:06:36.563090 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:06:36.564321 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:36.564955 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:36.566501 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:36.566812 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:36.566891 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:06:36.569247 (+ 2356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:06:36.569248 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:06:36.569249 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:06:36.569256 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:06:36.570660 (+ 1404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:06:36.571073 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:06:36.571074 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:06:36.571075 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:06:36.571140 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:06:36.571394 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:06:36.571395 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:06:36.571396 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:06:36.571512 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:06:36.571586 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0810 04:06:36.571857 (+ 271us) server_negotiation.cc:299] Negotiation successful
0810 04:06:36.571939 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260810 04:06:36.572268 32285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:36.562256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:41015 (local address 127.31.32.2:58181)
0810 04:06:36.562443 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0810 04:06:36.562478 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0810 04:06:36.562948 (+ 470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:06:36.563141 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:06:36.563144 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:06:36.563953 (+ 809us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:06:36.564231 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:06:36.564238 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:06:36.565074 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:06:36.565075 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:06:36.566411 (+ 1336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:06:36.566414 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:06:36.566902 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:06:36.566903 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:06:36.566999 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:06:36.567530 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:06:36.567540 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:06:36.569149 (+ 1609us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:06:36.570762 (+ 1613us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:06:36.570769 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:06:36.570771 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:06:36.570995 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:06:36.571210 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:06:36.571211 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:06:36.571212 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:06:36.571324 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:06:36.571588 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:06:36.571596 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:06:36.571729 (+ 133us) client_negotiation.cc:769] Sending connection context
0810 04:06:36.571834 (+ 105us) client_negotiation.cc:240] Negotiation successful
0810 04:06:36.571987 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":110,"threads_started":1}
WARNING: no policy specified for kudu/127.31.32.3@KRBTEST.COM; defaulting to no policy
I20260810 04:06:36.572755 32283 heartbeater.cc:347] Connected to a master server at 127.31.32.62:41015
I20260810 04:06:36.572862 32283 heartbeater.cc:464] Registering TS with master...
I20260810 04:06:36.573042 32283 heartbeater.cc:511] Master 127.31.32.62:41015 requested a full tablet report, sending...
I20260810 04:06:36.573446 31927 ts_manager.cc:194] Registered new tserver with Master: 3bf4bbc3d28b40ecb89251365a12d969 (127.31.32.2:33771)
I20260810 04:06:36.573817 31927 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.32.2@KRBTEST.COM'} at 127.31.32.2:58181
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.32.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.32.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.3.keytab.
Entry for principal kudu/127.31.32.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.3.keytab.
I20260810 04:06:36.591027 31872 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskws89Io/build/release/bin/kudu
/tmp/dist-test-taskws89Io/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786334794676589-31872-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786334794676589-31872-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786334794676589-31872-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786334794676589-31872-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.32.3:0
--local_ip_for_outbound_sockets=127.31.32.3
--webserver_interface=127.31.32.3
--webserver_port=0
--tserver_master_addrs=127.31.32.62:41015
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.3.keytab
--principal=kudu/127.31.32.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.32.20:38041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786334794676589-31872-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786334794676589-31872-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786334794676589-31872-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:06:36.673077 32288 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:06:36.673313 32288 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:06:36.673364 32288 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:06:36.674484 32288 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:06:36.674552 32288 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:06:36.674598 32288 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.32.3
I20260810 04:06:36.676040 32288 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.32.20:38041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786334794676589-31872-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786334794676589-31872-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.3.keytab
--principal=kudu/127.31.32.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.32.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786334794676589-31872-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.32.3
--webserver_port=0
--tserver_master_addrs=127.31.32.62:41015
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32288
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.32.3
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786334794676589-31872-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:06:36.676334 32288 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:06:36.676560 32288 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:06:36.681689 32294 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:06:36.681727 32288 server_base.cc:1034] running on GCE node
W20260810 04:06:36.681677 32299 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:06:36.681692 32296 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:06:36.682124 32288 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:06:36.682389 32288 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:06:36.683539 32288 hybrid_clock.cc:648] HybridClock initialized: now 1786334796683524 us; error 28 us; skew 500 ppm
Aug 10 04:06:36 dist-test-slave-8w9v krb5kdc[31881](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334796, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:06:36.686110 32288 init.cc:377] Logged in from keytab as kudu/127.31.32.3@KRBTEST.COM (short username kudu)
I20260810 04:06:36.686563 32288 webserver.cc:460] Webserver started at http://127.31.32.3:41425/ using document root <none> and password file <none>
I20260810 04:06:36.686830 32288 fs_manager.cc:359] Metadata directory not provided
I20260810 04:06:36.686911 32288 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:06:36.687062 32288 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:06:36.688097 32288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786334794676589-31872-0/minicluster-data/ts-2/data/instance:
uuid: "0ed67f959c14487b957f64478e5efd7b"
format_stamp: "Formatted at 2026-08-10 04:06:36 on dist-test-slave-8w9v"
server_key: "de3eaf51c702d33d8445985885111aaf"
server_key_iv: "c8e94da3251a46d8bb8f6d0b8450a234"
server_key_version: "encryptionkey@0"
I20260810 04:06:36.688524 32288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786334794676589-31872-0/minicluster-data/ts-2/wal/instance:
uuid: "0ed67f959c14487b957f64478e5efd7b"
format_stamp: "Formatted at 2026-08-10 04:06:36 on dist-test-slave-8w9v"
server_key: "de3eaf51c702d33d8445985885111aaf"
server_key_iv: "c8e94da3251a46d8bb8f6d0b8450a234"
server_key_version: "encryptionkey@0"
I20260810 04:06:36.689890 32288 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:06:36.690593 32306 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:06:36.690773 32288 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 04:06:36.690873 32288 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786334794676589-31872-0/minicluster-data/ts-2/data,/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786334794676589-31872-0/minicluster-data/ts-2/wal
uuid: "0ed67f959c14487b957f64478e5efd7b"
format_stamp: "Formatted at 2026-08-10 04:06:36 on dist-test-slave-8w9v"
server_key: "de3eaf51c702d33d8445985885111aaf"
server_key_iv: "c8e94da3251a46d8bb8f6d0b8450a234"
server_key_version: "encryptionkey@0"
I20260810 04:06:36.690955 32288 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786334794676589-31872-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786334794676589-31872-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786334794676589-31872-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:06:36.698899 32288 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:06:36.700668 32288 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:06:36.700834 32288 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:06:36.701053 32288 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:06:36.701350 32288 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:06:36.701404 32288 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:06:36.701435 32288 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:06:36.701462 32288 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:06:36.707756 32288 rpc_server.cc:307] RPC server started. Bound to: 127.31.32.3:40655
I20260810 04:06:36.707791 32434 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.32.3:40655 every 8 connection(s)
I20260810 04:06:36.708098 32288 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786334794676589-31872-0/minicluster-data/ts-2/data/info.pb
Aug 10 04:06:36 dist-test-slave-8w9v krb5kdc[31881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334796, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.3@KRBTEST.COM for kudu/127.31.32.62@KRBTEST.COM
I20260810 04:06:36.715658 31872 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskws89Io/build/release/bin/kudu as pid 32288
I20260810 04:06:36.715736 31872 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786334794676589-31872-0/minicluster-data/ts-2/wal/instance
I20260810 04:06:36.715899 31872 external_mini_cluster.cc:1428] Setting key f414857bed28f917ae6fb272af3b3085
I20260810 04:06:36.716964 31967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:36.708860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.3:57789 (local address 127.31.32.62:41015)
0810 04:06:36.708905 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0810 04:06:36.708906 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 04:06:36.709122 (+ 216us) server_negotiation.cc:407] Connection header received
0810 04:06:36.709695 (+ 573us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:06:36.709696 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:06:36.709710 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:06:36.709772 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:06:36.710704 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:36.711204 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:36.712156 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:36.712338 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:36.712392 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:06:36.714323 (+ 1931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:06:36.714324 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:06:36.714324 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:06:36.714330 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:06:36.715475 (+ 1145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:06:36.715951 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:06:36.715953 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:06:36.715953 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:06:36.716027 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:06:36.716312 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:06:36.716313 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:06:36.716313 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:06:36.716436 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:06:36.716514 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0810 04:06:36.716795 (+ 281us) server_negotiation.cc:299] Negotiation successful
0810 04:06:36.716875 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260810 04:06:36.717125 32437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:36.708835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:41015 (local address 127.31.32.3:57789)
0810 04:06:36.709003 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0810 04:06:36.709036 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0810 04:06:36.709589 (+ 553us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:06:36.709829 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:06:36.709833 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:06:36.710393 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:06:36.710571 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:06:36.710576 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:06:36.711293 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:06:36.711294 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:06:36.712052 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:06:36.712054 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:06:36.712412 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:06:36.712413 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:06:36.712516 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:06:36.712930 (+ 414us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:06:36.712938 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:06:36.714242 (+ 1304us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:06:36.715555 (+ 1313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:06:36.715563 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:06:36.715565 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:06:36.715863 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:06:36.716090 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:06:36.716091 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:06:36.716091 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:06:36.716246 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:06:36.716507 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:06:36.716514 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:06:36.716647 (+ 133us) client_negotiation.cc:769] Sending connection context
0810 04:06:36.716758 (+ 111us) client_negotiation.cc:240] Negotiation successful
0810 04:06:36.716905 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":82,"threads_started":1}
I20260810 04:06:36.717577 32435 heartbeater.cc:347] Connected to a master server at 127.31.32.62:41015
I20260810 04:06:36.717666 32435 heartbeater.cc:464] Registering TS with master...
I20260810 04:06:36.717849 32435 heartbeater.cc:511] Master 127.31.32.62:41015 requested a full tablet report, sending...
I20260810 04:06:36.718226 31927 ts_manager.cc:194] Registered new tserver with Master: 0ed67f959c14487b957f64478e5efd7b (127.31.32.3:40655)
I20260810 04:06:36.718549 31927 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.32.3@KRBTEST.COM'} at 127.31.32.3:57789
I20260810 04:06:36.728165 31872 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 10 04:06:36 dist-test-slave-8w9v krb5kdc[31881](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334796, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 10 04:06:36 dist-test-slave-8w9v krb5kdc[31881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334796, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.31.32.1@KRBTEST.COM
I20260810 04:06:36.751240 32442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:36.743473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47596 (local address 127.31.32.1:44177)
0810 04:06:36.743644 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0810 04:06:36.743648 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 04:06:36.743692 (+ 44us) server_negotiation.cc:407] Connection header received
0810 04:06:36.743721 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:06:36.743723 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:06:36.743739 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:06:36.743802 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:06:36.744465 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:36.745399 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:36.746076 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:36.746370 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:36.746421 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:06:36.748850 (+ 2429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:06:36.748858 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:06:36.748858 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:06:36.748865 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:06:36.750239 (+ 1374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:06:36.750549 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:06:36.750552 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:06:36.750552 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:06:36.750603 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:06:36.750764 (+ 161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:06:36.750765 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:06:36.750765 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:06:36.750890 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:06:36.750988 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0810 04:06:36.751052 (+ 64us) server_negotiation.cc:299] Negotiation successful
0810 04:06:36.751108 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":94,"threads_started":1}
W20260810 04:06:36.751842 32043 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47596
Aug 10 04:06:36 dist-test-slave-8w9v krb5kdc[31881](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334796, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 10 04:06:36 dist-test-slave-8w9v krb5kdc[31881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334796, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.32.1@KRBTEST.COM
I20260810 04:06:36.774055 32442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:36.767287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47606 (local address 127.31.32.1:44177)
0810 04:06:36.767375 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0810 04:06:36.767377 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:06:36.767390 (+ 13us) server_negotiation.cc:407] Connection header received
0810 04:06:36.767464 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:06:36.767465 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:06:36.767477 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:06:36.767532 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:06:36.768047 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:36.768614 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:36.769345 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:36.769547 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:36.769622 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:06:36.771709 (+ 2087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:06:36.771711 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:06:36.771711 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:06:36.771718 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:06:36.773194 (+ 1476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:06:36.773478 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:06:36.773478 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:06:36.773479 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:06:36.773528 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:06:36.773670 (+ 142us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:06:36.773670 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:06:36.773671 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:06:36.773747 (+ 76us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:06:36.773830 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0810 04:06:36.773894 (+ 64us) server_negotiation.cc:299] Negotiation successful
0810 04:06:36.773952 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 04:06:36.774626 31872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskws89Io/build/release/bin/kudu with pid 31996
I20260810 04:06:36.782153 31872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskws89Io/build/release/bin/kudu with pid 32144
I20260810 04:06:36.789229 31872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskws89Io/build/release/bin/kudu with pid 32288
I20260810 04:06:36.797197 31872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskws89Io/build/release/bin/kudu with pid 31894
2026-08-10T04:06:36Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2118 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786334794676589-31872-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 04:06:36 dist-test-slave-8w9v krb5kdc[32451](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 04:06:36 dist-test-slave-8w9v krb5kdc[32451](info): set up 2 sockets
Aug 10 04:06:36 dist-test-slave-8w9v krb5kdc[32451](info): commencing operation
krb5kdc: starting...
W20260810 04:06:37.882965 31872 mini_kdc.cc:121] Time spent starting KDC: real 1.064s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786334794676589-31872-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786334794676589-31872-0/krb5kdc/test-user.keytab.
Aug 10 04:06:37 dist-test-slave-8w9v krb5kdc[32451](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334797, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T04:06:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T04:06:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.32.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.32.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.32.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.62.keytab.
Entry for principal kudu/127.31.32.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.62.keytab.
I20260810 04:06:38.004931 31872 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskws89Io/build/release/bin/kudu
/tmp/dist-test-taskws89Io/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786334794676589-31872-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786334794676589-31872-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786334794676589-31872-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786334794676589-31872-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.32.62:46749
--webserver_interface=127.31.32.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.62.keytab
--principal=kudu/127.31.32.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.32.20:45045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.32.62:46749
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786334794676589-31872-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786334794676589-31872-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786334794676589-31872-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:06:38.087271 32466 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:06:38.087520 32466 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:06:38.087555 32466 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:06:38.088698 32466 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:06:38.088768 32466 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:06:38.088792 32466 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:06:38.088812 32466 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:06:38.088843 32466 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:06:38.090292 32466 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.32.20:45045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786334794676589-31872-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786334794676589-31872-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.32.62:46749
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.62.keytab
--principal=kudu/127.31.32.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.32.62:46749
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786334794676589-31872-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.32.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32466
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786334794676589-31872-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:06:38.090551 32466 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:06:38.090771 32466 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:06:38.095608 32471 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:06:38.095616 32472 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:06:38.095649 32474 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:06:38.095940 32466 server_base.cc:1034] running on GCE node
I20260810 04:06:38.096241 32466 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:06:38.096545 32466 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:06:38.097690 32466 hybrid_clock.cc:648] HybridClock initialized: now 1786334798097679 us; error 30 us; skew 500 ppm
Aug 10 04:06:38 dist-test-slave-8w9v krb5kdc[32451](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334798, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:06:38.105154 32466 init.cc:377] Logged in from keytab as kudu/127.31.32.62@KRBTEST.COM (short username kudu)
I20260810 04:06:38.105636 32466 webserver.cc:460] Webserver started at http://127.31.32.62:39025/ using document root <none> and password file <none>
I20260810 04:06:38.105877 32466 fs_manager.cc:359] Metadata directory not provided
I20260810 04:06:38.105945 32466 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:06:38.106096 32466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:06:38.107151 32466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786334794676589-31872-0/minicluster-data/master-0/data/instance:
uuid: "35b6a53f04bb4b65b87cfb53d3aa01fe"
format_stamp: "Formatted at 2026-08-10 04:06:38 on dist-test-slave-8w9v"
server_key: "361c366b2c52de1b079a4622f7c1d4f7"
server_key_iv: "173782915220f5411e498c60335bccd6"
server_key_version: "encryptionkey@0"
I20260810 04:06:38.107535 32466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786334794676589-31872-0/minicluster-data/master-0/wal/instance:
uuid: "35b6a53f04bb4b65b87cfb53d3aa01fe"
format_stamp: "Formatted at 2026-08-10 04:06:38 on dist-test-slave-8w9v"
server_key: "361c366b2c52de1b079a4622f7c1d4f7"
server_key_iv: "173782915220f5411e498c60335bccd6"
server_key_version: "encryptionkey@0"
I20260810 04:06:38.108983 32466 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 04:06:38.109752 32482 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:06:38.109992 32466 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:06:38.110085 32466 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786334794676589-31872-0/minicluster-data/master-0/data,/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786334794676589-31872-0/minicluster-data/master-0/wal
uuid: "35b6a53f04bb4b65b87cfb53d3aa01fe"
format_stamp: "Formatted at 2026-08-10 04:06:38 on dist-test-slave-8w9v"
server_key: "361c366b2c52de1b079a4622f7c1d4f7"
server_key_iv: "173782915220f5411e498c60335bccd6"
server_key_version: "encryptionkey@0"
I20260810 04:06:38.110174 32466 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786334794676589-31872-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786334794676589-31872-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786334794676589-31872-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:06:38.129305 32466 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:06:38.131122 32466 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:06:38.131282 32466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:06:38.135318 32466 rpc_server.cc:307] RPC server started. Bound to: 127.31.32.62:46749
I20260810 04:06:38.135386 32539 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.32.62:46749 every 8 connection(s)
I20260810 04:06:38.135711 32466 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786334794676589-31872-0/minicluster-data/master-0/data/info.pb
I20260810 04:06:38.136656 32540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:06:38.138854 32540 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 35b6a53f04bb4b65b87cfb53d3aa01fe: Bootstrap starting.
I20260810 04:06:38.139391 32540 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 35b6a53f04bb4b65b87cfb53d3aa01fe: Neither blocks nor log segments found. Creating new log.
I20260810 04:06:38.139628 32540 log.cc:824] T 00000000000000000000000000000000 P 35b6a53f04bb4b65b87cfb53d3aa01fe: Log is configured to *not* fsync() on all Append() calls
I20260810 04:06:38.140321 31872 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskws89Io/build/release/bin/kudu as pid 32466
I20260810 04:06:38.140409 31872 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786334794676589-31872-0/minicluster-data/master-0/wal/instance
I20260810 04:06:38.140460 32540 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 35b6a53f04bb4b65b87cfb53d3aa01fe: No bootstrap required, opened a new log
I20260810 04:06:38.140585 31872 external_mini_cluster.cc:1428] Setting key 1c361c410678f4312db06c08ddebfedd
I20260810 04:06:38.142940 32540 raft_consensus.cc:348] T 00000000000000000000000000000000 P 35b6a53f04bb4b65b87cfb53d3aa01fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35b6a53f04bb4b65b87cfb53d3aa01fe" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 46749 } }
I20260810 04:06:38.143070 32540 raft_consensus.cc:374] T 00000000000000000000000000000000 P 35b6a53f04bb4b65b87cfb53d3aa01fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:06:38.143091 32540 raft_consensus.cc:729] T 00000000000000000000000000000000 P 35b6a53f04bb4b65b87cfb53d3aa01fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35b6a53f04bb4b65b87cfb53d3aa01fe, State: Initialized, Role: FOLLOWER
I20260810 04:06:38.143208 32540 consensus_queue.cc:260] T 00000000000000000000000000000000 P 35b6a53f04bb4b65b87cfb53d3aa01fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35b6a53f04bb4b65b87cfb53d3aa01fe" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 46749 } }
I20260810 04:06:38.143302 32540 raft_consensus.cc:388] T 00000000000000000000000000000000 P 35b6a53f04bb4b65b87cfb53d3aa01fe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 04:06:38.143354 32540 raft_consensus.cc:482] T 00000000000000000000000000000000 P 35b6a53f04bb4b65b87cfb53d3aa01fe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 04:06:38.143402 32540 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 35b6a53f04bb4b65b87cfb53d3aa01fe [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:06:38.143930 32540 raft_consensus.cc:504] T 00000000000000000000000000000000 P 35b6a53f04bb4b65b87cfb53d3aa01fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35b6a53f04bb4b65b87cfb53d3aa01fe" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 46749 } }
I20260810 04:06:38.144053 32540 leader_election.cc:302] T 00000000000000000000000000000000 P 35b6a53f04bb4b65b87cfb53d3aa01fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 35b6a53f04bb4b65b87cfb53d3aa01fe; no voters:
I20260810 04:06:38.144266 32540 leader_election.cc:288] T 00000000000000000000000000000000 P 35b6a53f04bb4b65b87cfb53d3aa01fe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 04:06:38.144286 32545 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 35b6a53f04bb4b65b87cfb53d3aa01fe [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:06:38.144400 32545 raft_consensus.cc:686] T 00000000000000000000000000000000 P 35b6a53f04bb4b65b87cfb53d3aa01fe [term 1 LEADER]: Becoming Leader. State: Replica: 35b6a53f04bb4b65b87cfb53d3aa01fe, State: Running, Role: LEADER
I20260810 04:06:38.144482 32545 consensus_queue.cc:237] T 00000000000000000000000000000000 P 35b6a53f04bb4b65b87cfb53d3aa01fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35b6a53f04bb4b65b87cfb53d3aa01fe" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 46749 } }
I20260810 04:06:38.144546 32540 sys_catalog.cc:564] T 00000000000000000000000000000000 P 35b6a53f04bb4b65b87cfb53d3aa01fe [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:06:38.144785 32545 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35b6a53f04bb4b65b87cfb53d3aa01fe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "35b6a53f04bb4b65b87cfb53d3aa01fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35b6a53f04bb4b65b87cfb53d3aa01fe" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 46749 } } }
I20260810 04:06:38.144793 32549 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35b6a53f04bb4b65b87cfb53d3aa01fe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 35b6a53f04bb4b65b87cfb53d3aa01fe. Latest consensus state: current_term: 1 leader_uuid: "35b6a53f04bb4b65b87cfb53d3aa01fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35b6a53f04bb4b65b87cfb53d3aa01fe" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 46749 } } }
I20260810 04:06:38.144876 32545 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35b6a53f04bb4b65b87cfb53d3aa01fe [sys.catalog]: This master's current role is: LEADER
I20260810 04:06:38.144922 32549 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35b6a53f04bb4b65b87cfb53d3aa01fe [sys.catalog]: This master's current role is: LEADER
I20260810 04:06:38.145850 32554 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 04:06:38.146407 32554 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 04:06:38.147805 32554 catalog_manager.cc:1347] Generated new cluster ID: 7c9fbed02fa44d7f860b9eb03f5c3de6
I20260810 04:06:38.147857 32554 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 10 04:06:38 dist-test-slave-8w9v krb5kdc[32451](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334797, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.32.62@KRBTEST.COM
I20260810 04:06:38.153169 32544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:38.141199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57108 (local address 127.31.32.62:46749)
0810 04:06:38.141387 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0810 04:06:38.141405 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0810 04:06:38.141430 (+ 25us) server_negotiation.cc:407] Connection header received
0810 04:06:38.141828 (+ 398us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:06:38.141834 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:06:38.141879 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:06:38.141985 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:06:38.142802 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:38.144417 (+ 1615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:38.145537 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:38.146158 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:38.146271 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:06:38.149110 (+ 2839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:06:38.149114 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:06:38.149114 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:06:38.149125 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:06:38.151115 (+ 1990us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:06:38.151524 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:06:38.151527 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:06:38.151527 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:06:38.151626 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:06:38.151858 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:06:38.151859 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:06:38.151859 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:06:38.152190 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:06:38.152732 (+ 542us) server_negotiation.cc:1035] Waiting for connection context
0810 04:06:38.152826 (+ 94us) server_negotiation.cc:299] Negotiation successful
0810 04:06:38.152964 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":97,"threads_started":1}
I20260810 04:06:38.158015 32554 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 04:06:38.158704 32554 catalog_manager.cc:1504] Loading token signing keys...
I20260810 04:06:38.163089 32554 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 35b6a53f04bb4b65b87cfb53d3aa01fe: Generated new TSK 0
I20260810 04:06:38.163242 32554 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.32.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.32.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.32.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.1.keytab.
Entry for principal kudu/127.31.32.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.1.keytab.
I20260810 04:06:38.190714 31872 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskws89Io/build/release/bin/kudu
/tmp/dist-test-taskws89Io/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786334794676589-31872-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786334794676589-31872-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786334794676589-31872-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786334794676589-31872-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.32.1:0
--local_ip_for_outbound_sockets=127.31.32.1
--webserver_interface=127.31.32.1
--webserver_port=0
--tserver_master_addrs=127.31.32.62:46749
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.1.keytab
--principal=kudu/127.31.32.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.32.20:45045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786334794676589-31872-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786334794676589-31872-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786334794676589-31872-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:06:38.273984 32569 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:06:38.274221 32569 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:06:38.274263 32569 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:06:38.275475 32569 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:06:38.275542 32569 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:06:38.275590 32569 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.32.1
I20260810 04:06:38.277205 32569 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.32.20:45045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786334794676589-31872-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786334794676589-31872-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.1.keytab
--principal=kudu/127.31.32.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.32.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786334794676589-31872-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.32.1
--webserver_port=0
--tserver_master_addrs=127.31.32.62:46749
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32569
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.32.1
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786334794676589-31872-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:06:38.277506 32569 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:06:38.277740 32569 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:06:38.282840 32575 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:06:38.282842 32574 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:06:38.282896 32569 server_base.cc:1034] running on GCE node
W20260810 04:06:38.283138 32577 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:06:38.283430 32569 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:06:38.283676 32569 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:06:38.284844 32569 hybrid_clock.cc:648] HybridClock initialized: now 1786334798284837 us; error 37 us; skew 500 ppm
Aug 10 04:06:38 dist-test-slave-8w9v krb5kdc[32451](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334798, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:06:38.287842 32569 init.cc:377] Logged in from keytab as kudu/127.31.32.1@KRBTEST.COM (short username kudu)
I20260810 04:06:38.288352 32569 webserver.cc:460] Webserver started at http://127.31.32.1:39981/ using document root <none> and password file <none>
I20260810 04:06:38.288604 32569 fs_manager.cc:359] Metadata directory not provided
I20260810 04:06:38.288647 32569 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:06:38.288795 32569 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:06:38.289839 32569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786334794676589-31872-0/minicluster-data/ts-0/data/instance:
uuid: "76e8b1191b2441a59ae425125f636656"
format_stamp: "Formatted at 2026-08-10 04:06:38 on dist-test-slave-8w9v"
server_key: "571a07f349e36984cbea6b11e2937f06"
server_key_iv: "e433bc901b058c855575bf82694f827b"
server_key_version: "encryptionkey@0"
I20260810 04:06:38.290196 32569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786334794676589-31872-0/minicluster-data/ts-0/wal/instance:
uuid: "76e8b1191b2441a59ae425125f636656"
format_stamp: "Formatted at 2026-08-10 04:06:38 on dist-test-slave-8w9v"
server_key: "571a07f349e36984cbea6b11e2937f06"
server_key_iv: "e433bc901b058c855575bf82694f827b"
server_key_version: "encryptionkey@0"
I20260810 04:06:38.291561 32569 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 04:06:38.292356 32585 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:06:38.292572 32569 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:06:38.292677 32569 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786334794676589-31872-0/minicluster-data/ts-0/data,/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786334794676589-31872-0/minicluster-data/ts-0/wal
uuid: "76e8b1191b2441a59ae425125f636656"
format_stamp: "Formatted at 2026-08-10 04:06:38 on dist-test-slave-8w9v"
server_key: "571a07f349e36984cbea6b11e2937f06"
server_key_iv: "e433bc901b058c855575bf82694f827b"
server_key_version: "encryptionkey@0"
I20260810 04:06:38.292749 32569 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786334794676589-31872-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786334794676589-31872-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786334794676589-31872-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:06:38.307617 32569 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:06:38.309613 32569 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:06:38.309777 32569 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:06:38.310032 32569 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:06:38.310357 32569 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:06:38.310431 32569 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:06:38.310482 32569 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:06:38.310511 32569 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:06:38.317680 32569 rpc_server.cc:307] RPC server started. Bound to: 127.31.32.1:41127
I20260810 04:06:38.317776 32712 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.32.1:41127 every 8 connection(s)
I20260810 04:06:38.318089 32569 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786334794676589-31872-0/minicluster-data/ts-0/data/info.pb
Aug 10 04:06:38 dist-test-slave-8w9v krb5kdc[32451](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334798, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.1@KRBTEST.COM for kudu/127.31.32.62@KRBTEST.COM
I20260810 04:06:38.326100 31872 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskws89Io/build/release/bin/kudu as pid 32569
I20260810 04:06:38.326203 31872 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786334794676589-31872-0/minicluster-data/ts-0/wal/instance
I20260810 04:06:38.326390 31872 external_mini_cluster.cc:1428] Setting key 7d302dd963c943aee1c0413bc8b9552c
I20260810 04:06:38.327652 32715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:38.318798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:46749 (local address 127.31.32.1:44041)
0810 04:06:38.318983 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0810 04:06:38.319017 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0810 04:06:38.320020 (+ 1003us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:06:38.320291 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:06:38.320295 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:06:38.320555 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:06:38.320735 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:06:38.320739 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:06:38.321432 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:06:38.321433 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:06:38.322226 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:06:38.322228 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:06:38.322638 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:06:38.322639 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:06:38.322769 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:06:38.323243 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:06:38.323251 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:06:38.324724 (+ 1473us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:06:38.326151 (+ 1427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:06:38.326158 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:06:38.326160 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:06:38.326446 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:06:38.326704 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:06:38.326705 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:06:38.326705 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:06:38.326850 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:06:38.327135 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:06:38.327143 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:06:38.327274 (+ 131us) client_negotiation.cc:769] Sending connection context
0810 04:06:38.327375 (+ 101us) client_negotiation.cc:240] Negotiation successful
0810 04:06:38.327492 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":86,"threads_started":1}
I20260810 04:06:38.327662 32544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:38.318825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.1:44041 (local address 127.31.32.62:46749)
0810 04:06:38.318873 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0810 04:06:38.318875 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:06:38.319103 (+ 228us) server_negotiation.cc:407] Connection header received
0810 04:06:38.320149 (+ 1046us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:06:38.320150 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:06:38.320163 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:06:38.320211 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:06:38.320832 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:38.321337 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:38.322318 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:38.322543 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:38.322627 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:06:38.324821 (+ 2194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:06:38.324823 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:06:38.324823 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:06:38.324831 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:06:38.326060 (+ 1229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:06:38.326543 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:06:38.326544 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:06:38.326545 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:06:38.326615 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:06:38.326905 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:06:38.326905 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:06:38.326906 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:06:38.327060 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:06:38.327122 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0810 04:06:38.327425 (+ 303us) server_negotiation.cc:299] Negotiation successful
0810 04:06:38.327521 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260810 04:06:38.328431 32713 heartbeater.cc:347] Connected to a master server at 127.31.32.62:46749
I20260810 04:06:38.328532 32713 heartbeater.cc:464] Registering TS with master...
I20260810 04:06:38.328727 32713 heartbeater.cc:511] Master 127.31.32.62:46749 requested a full tablet report, sending...
I20260810 04:06:38.329336 32501 ts_manager.cc:194] Registered new tserver with Master: 76e8b1191b2441a59ae425125f636656 (127.31.32.1:41127)
I20260810 04:06:38.330257 32501 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.32.1@KRBTEST.COM'} at 127.31.32.1:44041
WARNING: no policy specified for kudu/127.31.32.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.32.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.32.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.2.keytab.
Entry for principal kudu/127.31.32.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.2.keytab.
I20260810 04:06:38.350828 31872 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskws89Io/build/release/bin/kudu
/tmp/dist-test-taskws89Io/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786334794676589-31872-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786334794676589-31872-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786334794676589-31872-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786334794676589-31872-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.32.2:0
--local_ip_for_outbound_sockets=127.31.32.2
--webserver_interface=127.31.32.2
--webserver_port=0
--tserver_master_addrs=127.31.32.62:46749
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.2.keytab
--principal=kudu/127.31.32.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.32.20:45045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786334794676589-31872-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786334794676589-31872-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786334794676589-31872-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:06:38.434894 32718 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:06:38.435083 32718 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:06:38.435106 32718 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:06:38.436282 32718 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:06:38.436324 32718 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:06:38.436352 32718 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.32.2
I20260810 04:06:38.437837 32718 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.32.20:45045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786334794676589-31872-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786334794676589-31872-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.2.keytab
--principal=kudu/127.31.32.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.32.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786334794676589-31872-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.32.2
--webserver_port=0
--tserver_master_addrs=127.31.32.62:46749
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32718
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.32.2
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786334794676589-31872-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:06:38.438098 32718 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:06:38.438344 32718 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:06:38.443567 32726 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:06:38.443585 32724 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:06:38.443602 32723 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:06:38.443722 32718 server_base.cc:1034] running on GCE node
I20260810 04:06:38.444016 32718 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:06:38.444298 32718 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:06:38.445439 32718 hybrid_clock.cc:648] HybridClock initialized: now 1786334798445422 us; error 34 us; skew 500 ppm
Aug 10 04:06:38 dist-test-slave-8w9v krb5kdc[32451](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334798, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:06:38.447958 32718 init.cc:377] Logged in from keytab as kudu/127.31.32.2@KRBTEST.COM (short username kudu)
I20260810 04:06:38.448478 32718 webserver.cc:460] Webserver started at http://127.31.32.2:34527/ using document root <none> and password file <none>
I20260810 04:06:38.448731 32718 fs_manager.cc:359] Metadata directory not provided
I20260810 04:06:38.448777 32718 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:06:38.448940 32718 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:06:38.450027 32718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786334794676589-31872-0/minicluster-data/ts-1/data/instance:
uuid: "bf8e107e00c34bd993ac943cee4e9c65"
format_stamp: "Formatted at 2026-08-10 04:06:38 on dist-test-slave-8w9v"
server_key: "5ad5d2c79476b730aa4654d23ba9f740"
server_key_iv: "2394b347858ac67f368f036fab173867"
server_key_version: "encryptionkey@0"
I20260810 04:06:38.450440 32718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786334794676589-31872-0/minicluster-data/ts-1/wal/instance:
uuid: "bf8e107e00c34bd993ac943cee4e9c65"
format_stamp: "Formatted at 2026-08-10 04:06:38 on dist-test-slave-8w9v"
server_key: "5ad5d2c79476b730aa4654d23ba9f740"
server_key_iv: "2394b347858ac67f368f036fab173867"
server_key_version: "encryptionkey@0"
I20260810 04:06:38.451774 32718 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 04:06:38.452620 32734 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:06:38.452836 32718 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260810 04:06:38.452937 32718 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786334794676589-31872-0/minicluster-data/ts-1/data,/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786334794676589-31872-0/minicluster-data/ts-1/wal
uuid: "bf8e107e00c34bd993ac943cee4e9c65"
format_stamp: "Formatted at 2026-08-10 04:06:38 on dist-test-slave-8w9v"
server_key: "5ad5d2c79476b730aa4654d23ba9f740"
server_key_iv: "2394b347858ac67f368f036fab173867"
server_key_version: "encryptionkey@0"
I20260810 04:06:38.453009 32718 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786334794676589-31872-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786334794676589-31872-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786334794676589-31872-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:06:38.462951 32718 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:06:38.464898 32718 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:06:38.465056 32718 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:06:38.465287 32718 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:06:38.465600 32718 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:06:38.465633 32718 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:06:38.465674 32718 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:06:38.465703 32718 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:06:38.471791 32718 rpc_server.cc:307] RPC server started. Bound to: 127.31.32.2:45093
I20260810 04:06:38.471839 398 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.32.2:45093 every 8 connection(s)
I20260810 04:06:38.472189 32718 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786334794676589-31872-0/minicluster-data/ts-1/data/info.pb
I20260810 04:06:38.475364 31872 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskws89Io/build/release/bin/kudu as pid 32718
I20260810 04:06:38.475442 31872 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786334794676589-31872-0/minicluster-data/ts-1/wal/instance
I20260810 04:06:38.475622 31872 external_mini_cluster.cc:1428] Setting key 70fff8edbe5c9d1a806c7ef81183dd6a
Aug 10 04:06:38 dist-test-slave-8w9v krb5kdc[32451](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334798, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.2@KRBTEST.COM for kudu/127.31.32.62@KRBTEST.COM
I20260810 04:06:38.482707 401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:38.472903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:46749 (local address 127.31.32.2:50305)
0810 04:06:38.473078 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0810 04:06:38.473112 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0810 04:06:38.474049 (+ 937us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:06:38.474327 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:06:38.474330 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:06:38.474600 (+ 270us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:06:38.474779 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:06:38.474783 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:06:38.475538 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:06:38.475539 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:06:38.476846 (+ 1307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:06:38.476849 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:06:38.477200 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:06:38.477201 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:06:38.477353 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:06:38.477823 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:06:38.477833 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:06:38.479455 (+ 1622us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:06:38.481364 (+ 1909us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:06:38.481370 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:06:38.481372 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:06:38.481585 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:06:38.481806 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:06:38.481807 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:06:38.481807 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:06:38.481908 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:06:38.482210 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:06:38.482215 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:06:38.482352 (+ 137us) client_negotiation.cc:769] Sending connection context
0810 04:06:38.482433 (+ 81us) client_negotiation.cc:240] Negotiation successful
0810 04:06:38.482550 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":84,"threads_started":1}
I20260810 04:06:38.482710 32544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:38.472939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.2:50305 (local address 127.31.32.62:46749)
0810 04:06:38.472984 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0810 04:06:38.472986 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:06:38.473194 (+ 208us) server_negotiation.cc:407] Connection header received
0810 04:06:38.474188 (+ 994us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:06:38.474189 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:06:38.474202 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:06:38.474260 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:06:38.474882 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:38.475410 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:38.476931 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:38.477115 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:38.477170 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:06:38.479551 (+ 2381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:06:38.479553 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:06:38.479553 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:06:38.479561 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:06:38.481279 (+ 1718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:06:38.481663 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:06:38.481665 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:06:38.481665 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:06:38.481733 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:06:38.481977 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:06:38.481978 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:06:38.481978 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:06:38.482137 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:06:38.482217 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0810 04:06:38.482487 (+ 270us) server_negotiation.cc:299] Negotiation successful
0810 04:06:38.482578 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260810 04:06:38.483387 399 heartbeater.cc:347] Connected to a master server at 127.31.32.62:46749
I20260810 04:06:38.483498 399 heartbeater.cc:464] Registering TS with master...
I20260810 04:06:38.483701 399 heartbeater.cc:511] Master 127.31.32.62:46749 requested a full tablet report, sending...
I20260810 04:06:38.484099 32501 ts_manager.cc:194] Registered new tserver with Master: bf8e107e00c34bd993ac943cee4e9c65 (127.31.32.2:45093)
I20260810 04:06:38.484519 32501 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.32.2@KRBTEST.COM'} at 127.31.32.2:50305
WARNING: no policy specified for kudu/127.31.32.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.32.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.32.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.3.keytab.
Entry for principal kudu/127.31.32.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.3.keytab.
I20260810 04:06:38.503041 31872 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskws89Io/build/release/bin/kudu
/tmp/dist-test-taskws89Io/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786334794676589-31872-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786334794676589-31872-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786334794676589-31872-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786334794676589-31872-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.32.3:0
--local_ip_for_outbound_sockets=127.31.32.3
--webserver_interface=127.31.32.3
--webserver_port=0
--tserver_master_addrs=127.31.32.62:46749
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.3.keytab
--principal=kudu/127.31.32.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.32.20:45045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786334794676589-31872-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786334794676589-31872-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786334794676589-31872-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:06:38.585391 404 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:06:38.585624 404 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:06:38.585665 404 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:06:38.586831 404 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:06:38.586900 404 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:06:38.586948 404 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.32.3
I20260810 04:06:38.588567 404 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.32.20:45045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786334794676589-31872-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786334794676589-31872-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.3.keytab
--principal=kudu/127.31.32.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.32.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786334794676589-31872-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.32.3
--webserver_port=0
--tserver_master_addrs=127.31.32.62:46749
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.404
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.32.3
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786334794676589-31872-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:06:38.588846 404 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:06:38.589080 404 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:06:38.594043 409 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:06:38.594050 410 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:06:38.594256 404 server_base.cc:1034] running on GCE node
W20260810 04:06:38.594120 413 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:06:38.594576 404 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:06:38.594825 404 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:06:38.596029 404 hybrid_clock.cc:648] HybridClock initialized: now 1786334798596014 us; error 27 us; skew 500 ppm
Aug 10 04:06:38 dist-test-slave-8w9v krb5kdc[32451](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334798, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:06:38.598518 404 init.cc:377] Logged in from keytab as kudu/127.31.32.3@KRBTEST.COM (short username kudu)
I20260810 04:06:38.598981 404 webserver.cc:460] Webserver started at http://127.31.32.3:42575/ using document root <none> and password file <none>
I20260810 04:06:38.599205 404 fs_manager.cc:359] Metadata directory not provided
I20260810 04:06:38.599248 404 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:06:38.599393 404 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:06:38.600515 404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786334794676589-31872-0/minicluster-data/ts-2/data/instance:
uuid: "d0a712f4fd04478383b5992a5cba9518"
format_stamp: "Formatted at 2026-08-10 04:06:38 on dist-test-slave-8w9v"
server_key: "5f1b02bbaf9e0ce72f0c26ae279b49b5"
server_key_iv: "0ffe2b53f5c4bb5abc925dfaf9b7fea7"
server_key_version: "encryptionkey@0"
I20260810 04:06:38.600883 404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786334794676589-31872-0/minicluster-data/ts-2/wal/instance:
uuid: "d0a712f4fd04478383b5992a5cba9518"
format_stamp: "Formatted at 2026-08-10 04:06:38 on dist-test-slave-8w9v"
server_key: "5f1b02bbaf9e0ce72f0c26ae279b49b5"
server_key_iv: "0ffe2b53f5c4bb5abc925dfaf9b7fea7"
server_key_version: "encryptionkey@0"
I20260810 04:06:38.602196 404 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 04:06:38.602866 420 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:06:38.603036 404 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 04:06:38.603137 404 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786334794676589-31872-0/minicluster-data/ts-2/data,/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786334794676589-31872-0/minicluster-data/ts-2/wal
uuid: "d0a712f4fd04478383b5992a5cba9518"
format_stamp: "Formatted at 2026-08-10 04:06:38 on dist-test-slave-8w9v"
server_key: "5f1b02bbaf9e0ce72f0c26ae279b49b5"
server_key_iv: "0ffe2b53f5c4bb5abc925dfaf9b7fea7"
server_key_version: "encryptionkey@0"
I20260810 04:06:38.603209 404 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786334794676589-31872-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786334794676589-31872-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786334794676589-31872-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:06:38.614531 404 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:06:38.616395 404 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:06:38.616545 404 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:06:38.616755 404 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:06:38.617060 404 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:06:38.617112 404 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:06:38.617153 404 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:06:38.617182 404 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:06:38.622991 404 rpc_server.cc:307] RPC server started. Bound to: 127.31.32.3:37879
I20260810 04:06:38.623029 546 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.32.3:37879 every 8 connection(s)
I20260810 04:06:38.623364 404 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786334794676589-31872-0/minicluster-data/ts-2/data/info.pb
I20260810 04:06:38.627921 31872 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskws89Io/build/release/bin/kudu as pid 404
I20260810 04:06:38.628028 31872 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786334794676589-31872-0/minicluster-data/ts-2/wal/instance
I20260810 04:06:38.628233 31872 external_mini_cluster.cc:1428] Setting key 7531289185b426cd05260c840db1639f
Aug 10 04:06:38 dist-test-slave-8w9v krb5kdc[32451](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334798, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.3@KRBTEST.COM for kudu/127.31.32.62@KRBTEST.COM
I20260810 04:06:38.632203 32544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:38.624075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.3:53953 (local address 127.31.32.62:46749)
0810 04:06:38.624201 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0810 04:06:38.624204 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:06:38.624365 (+ 161us) server_negotiation.cc:407] Connection header received
0810 04:06:38.625274 (+ 909us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:06:38.625275 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:06:38.625285 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:06:38.625335 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:06:38.625891 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:38.626365 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:38.627270 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:38.627443 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:38.627490 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:06:38.629810 (+ 2320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:06:38.629811 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:06:38.629811 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:06:38.629817 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:06:38.631037 (+ 1220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:06:38.631398 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:06:38.631399 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:06:38.631399 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:06:38.631447 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:06:38.631657 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:06:38.631658 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:06:38.631658 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:06:38.631752 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:06:38.631802 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0810 04:06:38.632007 (+ 205us) server_negotiation.cc:299] Negotiation successful
0810 04:06:38.632064 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260810 04:06:38.632423 549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:38.624060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:46749 (local address 127.31.32.3:53953)
0810 04:06:38.624233 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0810 04:06:38.624274 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 04:06:38.625185 (+ 911us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:06:38.625406 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:06:38.625409 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:06:38.625643 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:06:38.625821 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:06:38.625825 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:06:38.626437 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:06:38.626437 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:06:38.627198 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:06:38.627200 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:06:38.627511 (+ 311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:06:38.627512 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:06:38.627605 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:06:38.628104 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:06:38.628116 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:06:38.629732 (+ 1616us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:06:38.631102 (+ 1370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:06:38.631108 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:06:38.631109 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:06:38.631333 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:06:38.631503 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:06:38.631504 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:06:38.631504 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:06:38.631603 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:06:38.631805 (+ 202us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:06:38.631809 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:06:38.631918 (+ 109us) client_negotiation.cc:769] Sending connection context
0810 04:06:38.631995 (+ 77us) client_negotiation.cc:240] Negotiation successful
0810 04:06:38.632175 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":97,"threads_started":1}
I20260810 04:06:38.632860 547 heartbeater.cc:347] Connected to a master server at 127.31.32.62:46749
I20260810 04:06:38.632948 547 heartbeater.cc:464] Registering TS with master...
I20260810 04:06:38.633082 547 heartbeater.cc:511] Master 127.31.32.62:46749 requested a full tablet report, sending...
I20260810 04:06:38.633409 32501 ts_manager.cc:194] Registered new tserver with Master: d0a712f4fd04478383b5992a5cba9518 (127.31.32.3:37879)
I20260810 04:06:38.633769 32501 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.32.3@KRBTEST.COM'} at 127.31.32.3:53953
I20260810 04:06:38.640180 31872 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 04:06:38.648067 32544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:38.642014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57124 (local address 127.31.32.62:46749)
0810 04:06:38.642071 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0810 04:06:38.642073 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:06:38.642213 (+ 140us) server_negotiation.cc:407] Connection header received
0810 04:06:38.642276 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:06:38.642277 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:06:38.642285 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:06:38.642345 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:06:38.642906 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:38.643356 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:38.644061 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:38.644306 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:38.644369 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:06:38.645682 (+ 1313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:06:38.645683 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:06:38.645683 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:06:38.645688 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:06:38.647098 (+ 1410us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:06:38.647447 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:06:38.647448 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:06:38.647448 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:06:38.647497 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:06:38.647635 (+ 138us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:06:38.647636 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:06:38.647636 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:06:38.647751 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:06:38.647824 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0810 04:06:38.647932 (+ 108us) server_negotiation.cc:299] Negotiation successful
0810 04:06:38.647988 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260810 04:06:38.650538 32501 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57124:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 04:06:38.652278 32501 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 04:06:38.659227 557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:38.655158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.2:45093 (local address 127.0.0.1:52970)
0810 04:06:38.655751 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0810 04:06:38.655761 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 04:06:38.655818 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:06:38.656149 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:06:38.656150 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:06:38.656152 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:06:38.656198 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:06:38.656200 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:06:38.657215 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:06:38.657216 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:06:38.658186 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:06:38.658189 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:06:38.659020 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:06:38.659021 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:06:38.659075 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:06:38.659085 (+ 10us) client_negotiation.cc:769] Sending connection context
0810 04:06:38.659139 (+ 54us) client_negotiation.cc:240] Negotiation successful
0810 04:06:38.659176 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"mutex_wait_us":26,"thread_start_us":39,"threads_started":1}
I20260810 04:06:38.659294 562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:38.655565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52970 (local address 127.31.32.2:45093)
0810 04:06:38.655765 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0810 04:06:38.655771 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 04:06:38.655811 (+ 40us) server_negotiation.cc:407] Connection header received
0810 04:06:38.655890 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:06:38.655892 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:06:38.655907 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:06:38.655989 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:06:38.656371 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:38.657101 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:38.658277 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:38.658938 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:38.659014 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:06:38.659068 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0810 04:06:38.659173 (+ 105us) server_negotiation.cc:299] Negotiation successful
0810 04:06:38.659216 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":76,"threads_started":1}
I20260810 04:06:38.659713 558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:38.655115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.3:37879 (local address 127.0.0.1:35198)
0810 04:06:38.655280 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0810 04:06:38.655320 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0810 04:06:38.655388 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:06:38.655743 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:06:38.655746 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:06:38.655748 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:06:38.655825 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:06:38.655827 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:06:38.657185 (+ 1358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:06:38.657187 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:06:38.658503 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:06:38.658504 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:06:38.659480 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:06:38.659481 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:06:38.659546 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:06:38.659549 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 04:06:38.659606 (+ 57us) client_negotiation.cc:240] Negotiation successful
0810 04:06:38.659645 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"mutex_wait_us":69,"thread_start_us":49,"threads_started":1}
I20260810 04:06:38.659726 560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:38.655349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35198 (local address 127.31.32.3:37879)
0810 04:06:38.655545 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0810 04:06:38.655549 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 04:06:38.655567 (+ 18us) server_negotiation.cc:407] Connection header received
0810 04:06:38.655599 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:06:38.655600 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:06:38.655612 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:06:38.655711 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:06:38.655909 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:38.657078 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:38.658589 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:38.659312 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:38.659385 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:06:38.659443 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0810 04:06:38.659622 (+ 179us) server_negotiation.cc:299] Negotiation successful
0810 04:06:38.659658 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":81,"threads_started":1}
I20260810 04:06:38.660043 326 tablet_service.cc:1467] Processing CreateTablet for tablet 60ebe542c0d84d3d93cb5d0ab596ab16 (DEFAULT_TABLE table=test-table [id=e7140c22b5f748f08c66824a94c3e595]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:06:38.660483 326 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60ebe542c0d84d3d93cb5d0ab596ab16. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:06:38.660408 474 tablet_service.cc:1467] Processing CreateTablet for tablet 60ebe542c0d84d3d93cb5d0ab596ab16 (DEFAULT_TABLE table=test-table [id=e7140c22b5f748f08c66824a94c3e595]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:06:38.660710 474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60ebe542c0d84d3d93cb5d0ab596ab16. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:06:38.661175 559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:38.655144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.1:41127 (local address 127.0.0.1:46698)
0810 04:06:38.655335 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0810 04:06:38.655345 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 04:06:38.655389 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:06:38.656026 (+ 637us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:06:38.656027 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:06:38.656029 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:06:38.656070 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:06:38.656071 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:06:38.657442 (+ 1371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:06:38.657443 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:06:38.658501 (+ 1058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:06:38.658504 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:06:38.660941 (+ 2437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:06:38.660942 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:06:38.661002 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:06:38.661006 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 04:06:38.661070 (+ 64us) client_negotiation.cc:240] Negotiation successful
0810 04:06:38.661107 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":68,"threads_started":1}
I20260810 04:06:38.661218 561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:38.655360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46698 (local address 127.31.32.1:41127)
0810 04:06:38.655859 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0810 04:06:38.655863 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 04:06:38.655871 (+ 8us) server_negotiation.cc:407] Connection header received
0810 04:06:38.655908 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:06:38.655910 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:06:38.655924 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:06:38.656021 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:06:38.656219 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:38.657314 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:38.660049 (+ 2735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:38.660853 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:38.660928 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:06:38.660979 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0810 04:06:38.661095 (+ 116us) server_negotiation.cc:299] Negotiation successful
0810 04:06:38.661140 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":456,"thread_start_us":83,"threads_started":1}
I20260810 04:06:38.661896 32639 tablet_service.cc:1467] Processing CreateTablet for tablet 60ebe542c0d84d3d93cb5d0ab596ab16 (DEFAULT_TABLE table=test-table [id=e7140c22b5f748f08c66824a94c3e595]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:06:38.662228 32639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60ebe542c0d84d3d93cb5d0ab596ab16. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:06:38.663729 563 tablet_bootstrap.cc:492] T 60ebe542c0d84d3d93cb5d0ab596ab16 P d0a712f4fd04478383b5992a5cba9518: Bootstrap starting.
I20260810 04:06:38.664017 564 tablet_bootstrap.cc:492] T 60ebe542c0d84d3d93cb5d0ab596ab16 P bf8e107e00c34bd993ac943cee4e9c65: Bootstrap starting.
I20260810 04:06:38.664424 563 tablet_bootstrap.cc:654] T 60ebe542c0d84d3d93cb5d0ab596ab16 P d0a712f4fd04478383b5992a5cba9518: Neither blocks nor log segments found. Creating new log.
I20260810 04:06:38.664646 564 tablet_bootstrap.cc:654] T 60ebe542c0d84d3d93cb5d0ab596ab16 P bf8e107e00c34bd993ac943cee4e9c65: Neither blocks nor log segments found. Creating new log.
I20260810 04:06:38.665153 563 log.cc:824] T 60ebe542c0d84d3d93cb5d0ab596ab16 P d0a712f4fd04478383b5992a5cba9518: Log is configured to *not* fsync() on all Append() calls
I20260810 04:06:38.665218 564 log.cc:824] T 60ebe542c0d84d3d93cb5d0ab596ab16 P bf8e107e00c34bd993ac943cee4e9c65: Log is configured to *not* fsync() on all Append() calls
I20260810 04:06:38.665910 563 tablet_bootstrap.cc:492] T 60ebe542c0d84d3d93cb5d0ab596ab16 P d0a712f4fd04478383b5992a5cba9518: No bootstrap required, opened a new log
I20260810 04:06:38.665928 564 tablet_bootstrap.cc:492] T 60ebe542c0d84d3d93cb5d0ab596ab16 P bf8e107e00c34bd993ac943cee4e9c65: No bootstrap required, opened a new log
I20260810 04:06:38.665966 563 ts_tablet_manager.cc:1397] T 60ebe542c0d84d3d93cb5d0ab596ab16 P d0a712f4fd04478383b5992a5cba9518: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 04:06:38.666006 564 ts_tablet_manager.cc:1397] T 60ebe542c0d84d3d93cb5d0ab596ab16 P bf8e107e00c34bd993ac943cee4e9c65: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 04:06:38.667500 563 raft_consensus.cc:348] T 60ebe542c0d84d3d93cb5d0ab596ab16 P d0a712f4fd04478383b5992a5cba9518 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0a712f4fd04478383b5992a5cba9518" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 37879 } } peers { permanent_uuid: "76e8b1191b2441a59ae425125f636656" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 41127 } } peers { permanent_uuid: "bf8e107e00c34bd993ac943cee4e9c65" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 45093 } }
I20260810 04:06:38.667642 563 raft_consensus.cc:374] T 60ebe542c0d84d3d93cb5d0ab596ab16 P d0a712f4fd04478383b5992a5cba9518 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:06:38.667665 563 raft_consensus.cc:729] T 60ebe542c0d84d3d93cb5d0ab596ab16 P d0a712f4fd04478383b5992a5cba9518 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0a712f4fd04478383b5992a5cba9518, State: Initialized, Role: FOLLOWER
I20260810 04:06:38.667537 564 raft_consensus.cc:348] T 60ebe542c0d84d3d93cb5d0ab596ab16 P bf8e107e00c34bd993ac943cee4e9c65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0a712f4fd04478383b5992a5cba9518" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 37879 } } peers { permanent_uuid: "76e8b1191b2441a59ae425125f636656" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 41127 } } peers { permanent_uuid: "bf8e107e00c34bd993ac943cee4e9c65" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 45093 } }
I20260810 04:06:38.667725 564 raft_consensus.cc:374] T 60ebe542c0d84d3d93cb5d0ab596ab16 P bf8e107e00c34bd993ac943cee4e9c65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:06:38.667774 564 raft_consensus.cc:729] T 60ebe542c0d84d3d93cb5d0ab596ab16 P bf8e107e00c34bd993ac943cee4e9c65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf8e107e00c34bd993ac943cee4e9c65, State: Initialized, Role: FOLLOWER
I20260810 04:06:38.667788 563 consensus_queue.cc:260] T 60ebe542c0d84d3d93cb5d0ab596ab16 P d0a712f4fd04478383b5992a5cba9518 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0a712f4fd04478383b5992a5cba9518" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 37879 } } peers { permanent_uuid: "76e8b1191b2441a59ae425125f636656" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 41127 } } peers { permanent_uuid: "bf8e107e00c34bd993ac943cee4e9c65" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 45093 } }
I20260810 04:06:38.667899 564 consensus_queue.cc:260] T 60ebe542c0d84d3d93cb5d0ab596ab16 P bf8e107e00c34bd993ac943cee4e9c65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0a712f4fd04478383b5992a5cba9518" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 37879 } } peers { permanent_uuid: "76e8b1191b2441a59ae425125f636656" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 41127 } } peers { permanent_uuid: "bf8e107e00c34bd993ac943cee4e9c65" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 45093 } }
I20260810 04:06:38.668071 563 ts_tablet_manager.cc:1428] T 60ebe542c0d84d3d93cb5d0ab596ab16 P d0a712f4fd04478383b5992a5cba9518: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260810 04:06:38.668098 547 heartbeater.cc:503] Master 127.31.32.62:46749 was elected leader, sending a full tablet report...
I20260810 04:06:38.668179 564 ts_tablet_manager.cc:1428] T 60ebe542c0d84d3d93cb5d0ab596ab16 P bf8e107e00c34bd993ac943cee4e9c65: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260810 04:06:38.668295 399 heartbeater.cc:503] Master 127.31.32.62:46749 was elected leader, sending a full tablet report...
I20260810 04:06:38.668385 567 tablet_bootstrap.cc:492] T 60ebe542c0d84d3d93cb5d0ab596ab16 P 76e8b1191b2441a59ae425125f636656: Bootstrap starting.
I20260810 04:06:38.669092 567 tablet_bootstrap.cc:654] T 60ebe542c0d84d3d93cb5d0ab596ab16 P 76e8b1191b2441a59ae425125f636656: Neither blocks nor log segments found. Creating new log.
I20260810 04:06:38.669668 567 log.cc:824] T 60ebe542c0d84d3d93cb5d0ab596ab16 P 76e8b1191b2441a59ae425125f636656: Log is configured to *not* fsync() on all Append() calls
I20260810 04:06:38.670269 567 tablet_bootstrap.cc:492] T 60ebe542c0d84d3d93cb5d0ab596ab16 P 76e8b1191b2441a59ae425125f636656: No bootstrap required, opened a new log
I20260810 04:06:38.670320 567 ts_tablet_manager.cc:1397] T 60ebe542c0d84d3d93cb5d0ab596ab16 P 76e8b1191b2441a59ae425125f636656: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 04:06:38.671789 567 raft_consensus.cc:348] T 60ebe542c0d84d3d93cb5d0ab596ab16 P 76e8b1191b2441a59ae425125f636656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0a712f4fd04478383b5992a5cba9518" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 37879 } } peers { permanent_uuid: "76e8b1191b2441a59ae425125f636656" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 41127 } } peers { permanent_uuid: "bf8e107e00c34bd993ac943cee4e9c65" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 45093 } }
I20260810 04:06:38.671923 567 raft_consensus.cc:374] T 60ebe542c0d84d3d93cb5d0ab596ab16 P 76e8b1191b2441a59ae425125f636656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:06:38.671944 567 raft_consensus.cc:729] T 60ebe542c0d84d3d93cb5d0ab596ab16 P 76e8b1191b2441a59ae425125f636656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76e8b1191b2441a59ae425125f636656, State: Initialized, Role: FOLLOWER
I20260810 04:06:38.672015 567 consensus_queue.cc:260] T 60ebe542c0d84d3d93cb5d0ab596ab16 P 76e8b1191b2441a59ae425125f636656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0a712f4fd04478383b5992a5cba9518" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 37879 } } peers { permanent_uuid: "76e8b1191b2441a59ae425125f636656" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 41127 } } peers { permanent_uuid: "bf8e107e00c34bd993ac943cee4e9c65" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 45093 } }
I20260810 04:06:38.672303 32713 heartbeater.cc:503] Master 127.31.32.62:46749 was elected leader, sending a full tablet report...
I20260810 04:06:38.672286 567 ts_tablet_manager.cc:1428] T 60ebe542c0d84d3d93cb5d0ab596ab16 P 76e8b1191b2441a59ae425125f636656: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
W20260810 04:06:38.722728 400 tablet.cc:2369] T 60ebe542c0d84d3d93cb5d0ab596ab16 P bf8e107e00c34bd993ac943cee4e9c65: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 04:06:38.818799 32714 tablet.cc:2369] T 60ebe542c0d84d3d93cb5d0ab596ab16 P 76e8b1191b2441a59ae425125f636656: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 04:06:38.873970 548 tablet.cc:2369] T 60ebe542c0d84d3d93cb5d0ab596ab16 P d0a712f4fd04478383b5992a5cba9518: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 04:06:38.925714 568 raft_consensus.cc:482] T 60ebe542c0d84d3d93cb5d0ab596ab16 P d0a712f4fd04478383b5992a5cba9518 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 04:06:38.925850 568 raft_consensus.cc:504] T 60ebe542c0d84d3d93cb5d0ab596ab16 P d0a712f4fd04478383b5992a5cba9518 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0a712f4fd04478383b5992a5cba9518" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 37879 } } peers { permanent_uuid: "76e8b1191b2441a59ae425125f636656" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 41127 } } peers { permanent_uuid: "bf8e107e00c34bd993ac943cee4e9c65" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 45093 } }
I20260810 04:06:38.926532 568 leader_election.cc:288] T 60ebe542c0d84d3d93cb5d0ab596ab16 P d0a712f4fd04478383b5992a5cba9518 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76e8b1191b2441a59ae425125f636656 (127.31.32.1:41127), bf8e107e00c34bd993ac943cee4e9c65 (127.31.32.2:45093)
I20260810 04:06:38.929751 549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:38.926721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.1:41127 (local address 127.31.32.3:54981)
0810 04:06:38.926799 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0810 04:06:38.926815 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 04:06:38.926883 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:06:38.927137 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:06:38.927139 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:06:38.927140 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:06:38.927235 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:06:38.927252 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:06:38.928007 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:06:38.928008 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:06:38.928897 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:06:38.928899 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:06:38.929465 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:06:38.929466 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:06:38.929551 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:06:38.929556 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 04:06:38.929620 (+ 64us) client_negotiation.cc:240] Negotiation successful
0810 04:06:38.929671 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260810 04:06:38.929764 561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:38.926787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.3:54981 (local address 127.31.32.1:41127)
0810 04:06:38.926844 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0810 04:06:38.926846 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:06:38.926859 (+ 13us) server_negotiation.cc:407] Connection header received
0810 04:06:38.927020 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:06:38.927021 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:06:38.927033 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:06:38.927107 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:06:38.927378 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:38.927918 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:38.929000 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:38.929377 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:38.929432 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:06:38.929440 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0810 04:06:38.929634 (+ 194us) server_negotiation.cc:299] Negotiation successful
0810 04:06:38.929672 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260810 04:06:38.929888 572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:38.926816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.2:45093 (local address 127.31.32.3:40823)
0810 04:06:38.927031 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0810 04:06:38.927043 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 04:06:38.927100 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:06:38.927376 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:06:38.927377 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:06:38.927379 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:06:38.927449 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:06:38.927451 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:06:38.928161 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:06:38.928162 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:06:38.928932 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:06:38.928934 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:06:38.929586 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:06:38.929586 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:06:38.929685 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:06:38.929689 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 04:06:38.929748 (+ 59us) client_negotiation.cc:240] Negotiation successful
0810 04:06:38.929799 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":139,"threads_started":1}
I20260810 04:06:38.929916 562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:38.926898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.3:40823 (local address 127.31.32.2:45093)
0810 04:06:38.926971 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0810 04:06:38.926973 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:06:38.927118 (+ 145us) server_negotiation.cc:407] Connection header received
0810 04:06:38.927155 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:06:38.927156 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:06:38.927171 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:06:38.927283 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:06:38.927535 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:38.928036 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:38.929054 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:38.929496 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:38.929579 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:06:38.929591 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0810 04:06:38.929783 (+ 192us) server_negotiation.cc:299] Negotiation successful
0810 04:06:38.929831 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 04:06:38.930212 32661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60ebe542c0d84d3d93cb5d0ab596ab16" candidate_uuid: "d0a712f4fd04478383b5992a5cba9518" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76e8b1191b2441a59ae425125f636656" is_pre_election: true
I20260810 04:06:38.930511 32661 raft_consensus.cc:2454] T 60ebe542c0d84d3d93cb5d0ab596ab16 P 76e8b1191b2441a59ae425125f636656 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a712f4fd04478383b5992a5cba9518 in term 0.
I20260810 04:06:38.930384 348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60ebe542c0d84d3d93cb5d0ab596ab16" candidate_uuid: "d0a712f4fd04478383b5992a5cba9518" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf8e107e00c34bd993ac943cee4e9c65" is_pre_election: true
I20260810 04:06:38.930685 348 raft_consensus.cc:2454] T 60ebe542c0d84d3d93cb5d0ab596ab16 P bf8e107e00c34bd993ac943cee4e9c65 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a712f4fd04478383b5992a5cba9518 in term 0.
I20260810 04:06:38.930860 422 leader_election.cc:302] T 60ebe542c0d84d3d93cb5d0ab596ab16 P d0a712f4fd04478383b5992a5cba9518 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76e8b1191b2441a59ae425125f636656, d0a712f4fd04478383b5992a5cba9518; no voters:
I20260810 04:06:38.931051 568 raft_consensus.cc:2781] T 60ebe542c0d84d3d93cb5d0ab596ab16 P d0a712f4fd04478383b5992a5cba9518 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 04:06:38.931129 568 raft_consensus.cc:482] T 60ebe542c0d84d3d93cb5d0ab596ab16 P d0a712f4fd04478383b5992a5cba9518 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 04:06:38.931169 568 raft_consensus.cc:3037] T 60ebe542c0d84d3d93cb5d0ab596ab16 P d0a712f4fd04478383b5992a5cba9518 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:06:38.931928 568 raft_consensus.cc:504] T 60ebe542c0d84d3d93cb5d0ab596ab16 P d0a712f4fd04478383b5992a5cba9518 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0a712f4fd04478383b5992a5cba9518" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 37879 } } peers { permanent_uuid: "76e8b1191b2441a59ae425125f636656" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 41127 } } peers { permanent_uuid: "bf8e107e00c34bd993ac943cee4e9c65" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 45093 } }
I20260810 04:06:38.932096 568 leader_election.cc:288] T 60ebe542c0d84d3d93cb5d0ab596ab16 P d0a712f4fd04478383b5992a5cba9518 [CANDIDATE]: Term 1 election: Requested vote from peers 76e8b1191b2441a59ae425125f636656 (127.31.32.1:41127), bf8e107e00c34bd993ac943cee4e9c65 (127.31.32.2:45093)
I20260810 04:06:38.932340 348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60ebe542c0d84d3d93cb5d0ab596ab16" candidate_uuid: "d0a712f4fd04478383b5992a5cba9518" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf8e107e00c34bd993ac943cee4e9c65"
I20260810 04:06:38.932358 32661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60ebe542c0d84d3d93cb5d0ab596ab16" candidate_uuid: "d0a712f4fd04478383b5992a5cba9518" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76e8b1191b2441a59ae425125f636656"
I20260810 04:06:38.932423 348 raft_consensus.cc:3037] T 60ebe542c0d84d3d93cb5d0ab596ab16 P bf8e107e00c34bd993ac943cee4e9c65 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:06:38.932447 32661 raft_consensus.cc:3037] T 60ebe542c0d84d3d93cb5d0ab596ab16 P 76e8b1191b2441a59ae425125f636656 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:06:38.933158 348 raft_consensus.cc:2454] T 60ebe542c0d84d3d93cb5d0ab596ab16 P bf8e107e00c34bd993ac943cee4e9c65 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a712f4fd04478383b5992a5cba9518 in term 1.
I20260810 04:06:38.933168 32661 raft_consensus.cc:2454] T 60ebe542c0d84d3d93cb5d0ab596ab16 P 76e8b1191b2441a59ae425125f636656 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a712f4fd04478383b5992a5cba9518 in term 1.
I20260810 04:06:38.933393 422 leader_election.cc:302] T 60ebe542c0d84d3d93cb5d0ab596ab16 P d0a712f4fd04478383b5992a5cba9518 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76e8b1191b2441a59ae425125f636656, d0a712f4fd04478383b5992a5cba9518; no voters:
I20260810 04:06:38.933511 568 raft_consensus.cc:2781] T 60ebe542c0d84d3d93cb5d0ab596ab16 P d0a712f4fd04478383b5992a5cba9518 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:06:38.933681 568 raft_consensus.cc:686] T 60ebe542c0d84d3d93cb5d0ab596ab16 P d0a712f4fd04478383b5992a5cba9518 [term 1 LEADER]: Becoming Leader. State: Replica: d0a712f4fd04478383b5992a5cba9518, State: Running, Role: LEADER
I20260810 04:06:38.933797 568 consensus_queue.cc:237] T 60ebe542c0d84d3d93cb5d0ab596ab16 P d0a712f4fd04478383b5992a5cba9518 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0a712f4fd04478383b5992a5cba9518" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 37879 } } peers { permanent_uuid: "76e8b1191b2441a59ae425125f636656" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 41127 } } peers { permanent_uuid: "bf8e107e00c34bd993ac943cee4e9c65" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 45093 } }
I20260810 04:06:38.934844 32501 catalog_manager.cc:5495] T 60ebe542c0d84d3d93cb5d0ab596ab16 P d0a712f4fd04478383b5992a5cba9518 reported cstate change: term changed from 0 to 1, leader changed from <none> to d0a712f4fd04478383b5992a5cba9518 (127.31.32.3). New cstate: current_term: 1 leader_uuid: "d0a712f4fd04478383b5992a5cba9518" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0a712f4fd04478383b5992a5cba9518" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 37879 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "76e8b1191b2441a59ae425125f636656" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 41127 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf8e107e00c34bd993ac943cee4e9c65" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 45093 } health_report { overall_health: UNKNOWN } } }
Aug 10 04:06:38 dist-test-slave-8w9v krb5kdc[32451](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334797, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.32.1@KRBTEST.COM
I20260810 04:06:38.944252 561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:38.937039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46712 (local address 127.31.32.1:41127)
0810 04:06:38.937081 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0810 04:06:38.937083 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:06:38.937180 (+ 97us) server_negotiation.cc:407] Connection header received
0810 04:06:38.937233 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:06:38.937234 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:06:38.937241 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:06:38.937286 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:06:38.937800 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:38.938245 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:38.938904 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:38.939068 (+ 164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:38.939117 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:06:38.941411 (+ 2294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:06:38.941419 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:06:38.941419 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:06:38.941426 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:06:38.943103 (+ 1677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:06:38.943410 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:06:38.943413 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:06:38.943413 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:06:38.943475 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:06:38.943639 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:06:38.943639 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:06:38.943639 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:06:38.943762 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:06:38.943873 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0810 04:06:38.944018 (+ 145us) server_negotiation.cc:299] Negotiation successful
0810 04:06:38.944101 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
Aug 10 04:06:38 dist-test-slave-8w9v krb5kdc[32451](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334798, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 10 04:06:38 dist-test-slave-8w9v krb5kdc[32451](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334798, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.31.32.1@KRBTEST.COM
I20260810 04:06:38.967643 561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:38.961208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46722 (local address 127.31.32.1:41127)
0810 04:06:38.961285 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0810 04:06:38.961287 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:06:38.961330 (+ 43us) server_negotiation.cc:407] Connection header received
0810 04:06:38.961380 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:06:38.961381 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:06:38.961389 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:06:38.961438 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:06:38.961952 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:38.962413 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:38.963105 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:38.963277 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:38.963328 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:06:38.965432 (+ 2104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:06:38.965434 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:06:38.965434 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:06:38.965441 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:06:38.966643 (+ 1202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:06:38.966953 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:06:38.966953 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:06:38.966954 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:06:38.967003 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:06:38.967165 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:06:38.967166 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:06:38.967166 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:06:38.967257 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:06:38.967333 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0810 04:06:38.967463 (+ 130us) server_negotiation.cc:299] Negotiation successful
0810 04:06:38.967524 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
Aug 10 04:06:38 dist-test-slave-8w9v krb5kdc[32451](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334798, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 10 04:06:38 dist-test-slave-8w9v krb5kdc[32451](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334798, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.32.1@KRBTEST.COM
I20260810 04:06:38.989936 561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:38.982942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46734 (local address 127.31.32.1:41127)
0810 04:06:38.983014 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0810 04:06:38.983017 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:06:38.983076 (+ 59us) server_negotiation.cc:407] Connection header received
0810 04:06:38.983164 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:06:38.983165 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:06:38.983175 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:06:38.983251 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:06:38.983779 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:38.984302 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:38.985081 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:38.985266 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:38.985357 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:06:38.987511 (+ 2154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:06:38.987513 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:06:38.987513 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:06:38.987521 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:06:38.988821 (+ 1300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:06:38.989181 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:06:38.989182 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:06:38.989183 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:06:38.989241 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:06:38.989443 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:06:38.989444 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:06:38.989445 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:06:38.989543 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:06:38.989647 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0810 04:06:38.989751 (+ 104us) server_negotiation.cc:299] Negotiation successful
0810 04:06:38.989830 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
W20260810 04:06:38.990624 32619 tablet_service.cc:2908] Rejecting scan request for tablet 60ebe542c0d84d3d93cb5d0ab596ab16: Uninitialized: clean time has not yet been initialized
I20260810 04:06:38.991619 31872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskws89Io/build/release/bin/kudu with pid 32569
I20260810 04:06:39.000087 31872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskws89Io/build/release/bin/kudu with pid 32718
I20260810 04:06:39.006194 31872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskws89Io/build/release/bin/kudu with pid 404
I20260810 04:06:39.012444 31872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskws89Io/build/release/bin/kudu with pid 32466
2026-08-10T04:06:39Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2215 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786334794676589-31872-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 04:06:39 dist-test-slave-8w9v krb5kdc[589](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 04:06:39 dist-test-slave-8w9v krb5kdc[589](info): set up 2 sockets
Aug 10 04:06:39 dist-test-slave-8w9v krb5kdc[589](info): commencing operation
krb5kdc: starting...
W20260810 04:06:40.107410 31872 mini_kdc.cc:121] Time spent starting KDC: real 1.073s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786334794676589-31872-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786334794676589-31872-0/krb5kdc/test-user.keytab.
Aug 10 04:06:40 dist-test-slave-8w9v krb5kdc[589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334800, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T04:06:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T04:06:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.32.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.32.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.32.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.62.keytab.
Entry for principal kudu/127.31.32.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.62.keytab.
I20260810 04:06:40.222663 31872 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskws89Io/build/release/bin/kudu
/tmp/dist-test-taskws89Io/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786334794676589-31872-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786334794676589-31872-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786334794676589-31872-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786334794676589-31872-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.32.62:38035
--webserver_interface=127.31.32.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.62.keytab
--principal=kudu/127.31.32.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.32.20:41905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.32.62:38035
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786334794676589-31872-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786334794676589-31872-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786334794676589-31872-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:06:40.303558 603 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:06:40.303745 603 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:06:40.303766 603 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:06:40.304870 603 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:06:40.304905 603 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260810 04:06:40.304919 603 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:06:40.304932 603 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:06:40.304944 603 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:06:40.304956 603 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:06:40.306304 603 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.32.20:41905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786334794676589-31872-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786334794676589-31872-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.32.62:38035
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.62.keytab
--principal=kudu/127.31.32.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.32.62:38035
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786334794676589-31872-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.32.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.603
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786334794676589-31872-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:06:40.306500 603 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:06:40.306663 603 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:06:40.311381 612 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:06:40.311354 610 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:06:40.311416 603 server_base.cc:1034] running on GCE node
W20260810 04:06:40.311347 609 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:06:40.311707 603 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:06:40.312011 603 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:06:40.313190 603 hybrid_clock.cc:648] HybridClock initialized: now 1786334800313178 us; error 43 us; skew 500 ppm
Aug 10 04:06:40 dist-test-slave-8w9v krb5kdc[589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334800, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:06:40.315939 603 init.cc:377] Logged in from keytab as kudu/127.31.32.62@KRBTEST.COM (short username kudu)
I20260810 04:06:40.316431 603 webserver.cc:460] Webserver started at http://127.31.32.62:36979/ using document root <none> and password file <none>
I20260810 04:06:40.316695 603 fs_manager.cc:359] Metadata directory not provided
I20260810 04:06:40.316753 603 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:06:40.316936 603 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:06:40.317988 603 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786334794676589-31872-0/minicluster-data/master-0/data/instance:
uuid: "b898b856782d4f13872e6e2373d5a953"
format_stamp: "Formatted at 2026-08-10 04:06:40 on dist-test-slave-8w9v"
server_key: "eb478417d9bfca5dc5806940b821eae1"
server_key_iv: "9f293ffe9a05823181fe21d64d94465b"
server_key_version: "encryptionkey@0"
I20260810 04:06:40.318370 603 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786334794676589-31872-0/minicluster-data/master-0/wal/instance:
uuid: "b898b856782d4f13872e6e2373d5a953"
format_stamp: "Formatted at 2026-08-10 04:06:40 on dist-test-slave-8w9v"
server_key: "eb478417d9bfca5dc5806940b821eae1"
server_key_iv: "9f293ffe9a05823181fe21d64d94465b"
server_key_version: "encryptionkey@0"
I20260810 04:06:40.319718 603 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 04:06:40.320547 619 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:06:40.320745 603 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260810 04:06:40.320859 603 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786334794676589-31872-0/minicluster-data/master-0/data,/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786334794676589-31872-0/minicluster-data/master-0/wal
uuid: "b898b856782d4f13872e6e2373d5a953"
format_stamp: "Formatted at 2026-08-10 04:06:40 on dist-test-slave-8w9v"
server_key: "eb478417d9bfca5dc5806940b821eae1"
server_key_iv: "9f293ffe9a05823181fe21d64d94465b"
server_key_version: "encryptionkey@0"
I20260810 04:06:40.320932 603 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786334794676589-31872-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786334794676589-31872-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786334794676589-31872-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:06:40.329200 603 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:06:40.330848 603 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:06:40.330988 603 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:06:40.335984 603 rpc_server.cc:307] RPC server started. Bound to: 127.31.32.62:38035
I20260810 04:06:40.336076 692 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.32.62:38035 every 8 connection(s)
I20260810 04:06:40.336447 603 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786334794676589-31872-0/minicluster-data/master-0/data/info.pb
I20260810 04:06:40.337124 693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:06:40.337769 31872 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskws89Io/build/release/bin/kudu as pid 603
I20260810 04:06:40.337832 31872 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786334794676589-31872-0/minicluster-data/master-0/wal/instance
I20260810 04:06:40.338022 31872 external_mini_cluster.cc:1428] Setting key c16dae3df395e077efaa436a920bc0cb
I20260810 04:06:40.340127 693 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b898b856782d4f13872e6e2373d5a953: Bootstrap starting.
I20260810 04:06:40.340855 693 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b898b856782d4f13872e6e2373d5a953: Neither blocks nor log segments found. Creating new log.
I20260810 04:06:40.341161 693 log.cc:824] T 00000000000000000000000000000000 P b898b856782d4f13872e6e2373d5a953: Log is configured to *not* fsync() on all Append() calls
I20260810 04:06:40.341857 693 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b898b856782d4f13872e6e2373d5a953: No bootstrap required, opened a new log
I20260810 04:06:40.343737 693 raft_consensus.cc:348] T 00000000000000000000000000000000 P b898b856782d4f13872e6e2373d5a953 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b898b856782d4f13872e6e2373d5a953" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 38035 } }
I20260810 04:06:40.343914 693 raft_consensus.cc:374] T 00000000000000000000000000000000 P b898b856782d4f13872e6e2373d5a953 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:06:40.343966 693 raft_consensus.cc:729] T 00000000000000000000000000000000 P b898b856782d4f13872e6e2373d5a953 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b898b856782d4f13872e6e2373d5a953, State: Initialized, Role: FOLLOWER
I20260810 04:06:40.344094 693 consensus_queue.cc:260] T 00000000000000000000000000000000 P b898b856782d4f13872e6e2373d5a953 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b898b856782d4f13872e6e2373d5a953" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 38035 } }
I20260810 04:06:40.344197 693 raft_consensus.cc:388] T 00000000000000000000000000000000 P b898b856782d4f13872e6e2373d5a953 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 04:06:40.344233 693 raft_consensus.cc:482] T 00000000000000000000000000000000 P b898b856782d4f13872e6e2373d5a953 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 04:06:40.344283 693 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b898b856782d4f13872e6e2373d5a953 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:06:40.344771 693 raft_consensus.cc:504] T 00000000000000000000000000000000 P b898b856782d4f13872e6e2373d5a953 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b898b856782d4f13872e6e2373d5a953" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 38035 } }
I20260810 04:06:40.344857 693 leader_election.cc:302] T 00000000000000000000000000000000 P b898b856782d4f13872e6e2373d5a953 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b898b856782d4f13872e6e2373d5a953; no voters:
I20260810 04:06:40.345046 693 leader_election.cc:288] T 00000000000000000000000000000000 P b898b856782d4f13872e6e2373d5a953 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 04:06:40.345084 698 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b898b856782d4f13872e6e2373d5a953 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:06:40.345245 698 raft_consensus.cc:686] T 00000000000000000000000000000000 P b898b856782d4f13872e6e2373d5a953 [term 1 LEADER]: Becoming Leader. State: Replica: b898b856782d4f13872e6e2373d5a953, State: Running, Role: LEADER
I20260810 04:06:40.345305 693 sys_catalog.cc:564] T 00000000000000000000000000000000 P b898b856782d4f13872e6e2373d5a953 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:06:40.345377 698 consensus_queue.cc:237] T 00000000000000000000000000000000 P b898b856782d4f13872e6e2373d5a953 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b898b856782d4f13872e6e2373d5a953" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 38035 } }
Aug 10 04:06:40 dist-test-slave-8w9v krb5kdc[589](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334800, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.32.62@KRBTEST.COM
I20260810 04:06:40.345690 698 sys_catalog.cc:455] T 00000000000000000000000000000000 P b898b856782d4f13872e6e2373d5a953 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b898b856782d4f13872e6e2373d5a953. Latest consensus state: current_term: 1 leader_uuid: "b898b856782d4f13872e6e2373d5a953" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b898b856782d4f13872e6e2373d5a953" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 38035 } } }
I20260810 04:06:40.345789 698 sys_catalog.cc:458] T 00000000000000000000000000000000 P b898b856782d4f13872e6e2373d5a953 [sys.catalog]: This master's current role is: LEADER
I20260810 04:06:40.345701 699 sys_catalog.cc:455] T 00000000000000000000000000000000 P b898b856782d4f13872e6e2373d5a953 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b898b856782d4f13872e6e2373d5a953" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b898b856782d4f13872e6e2373d5a953" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 38035 } } }
I20260810 04:06:40.345953 699 sys_catalog.cc:458] T 00000000000000000000000000000000 P b898b856782d4f13872e6e2373d5a953 [sys.catalog]: This master's current role is: LEADER
I20260810 04:06:40.346019 705 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 04:06:40.346544 705 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 04:06:40.347803 715 mvcc.cc:205] Tried to move back new op lower bound from 7316827342223052800 to 7316827342215114752. Current Snapshot: MvccSnapshot[applied={T|T < 7316827342223052800}]
I20260810 04:06:40.348189 705 catalog_manager.cc:1347] Generated new cluster ID: b95ae7710466429dbadbe33d840fd7ed
I20260810 04:06:40.348237 705 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 04:06:40.350121 696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:40.338560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33390 (local address 127.31.32.62:38035)
0810 04:06:40.338744 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0810 04:06:40.338753 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 04:06:40.338786 (+ 33us) server_negotiation.cc:407] Connection header received
0810 04:06:40.339387 (+ 601us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:06:40.339396 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:06:40.339466 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:06:40.339609 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:06:40.340264 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:40.341592 (+ 1328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:40.342365 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:40.343620 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:40.343727 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:06:40.346313 (+ 2586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:06:40.346316 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:06:40.346317 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:06:40.346328 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:06:40.348545 (+ 2217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:06:40.348940 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:06:40.348943 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:06:40.348943 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:06:40.349046 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:06:40.349274 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:06:40.349275 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:06:40.349275 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:06:40.349569 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:06:40.349721 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0810 04:06:40.349788 (+ 67us) server_negotiation.cc:299] Negotiation successful
0810 04:06:40.349921 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":82,"threads_started":1}
I20260810 04:06:40.358072 705 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 04:06:40.358723 705 catalog_manager.cc:1504] Loading token signing keys...
I20260810 04:06:40.364423 705 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b898b856782d4f13872e6e2373d5a953: Generated new TSK 0
I20260810 04:06:40.364578 705 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.32.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.32.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.32.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.1.keytab.
Entry for principal kudu/127.31.32.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.1.keytab.
I20260810 04:06:40.394341 31872 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskws89Io/build/release/bin/kudu
/tmp/dist-test-taskws89Io/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786334794676589-31872-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786334794676589-31872-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786334794676589-31872-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786334794676589-31872-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.32.1:0
--local_ip_for_outbound_sockets=127.31.32.1
--webserver_interface=127.31.32.1
--webserver_port=0
--tserver_master_addrs=127.31.32.62:38035
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.1.keytab
--principal=kudu/127.31.32.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.32.20:41905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786334794676589-31872-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786334794676589-31872-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786334794676589-31872-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:06:40.474367 719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:06:40.474553 719 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:06:40.474576 719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:06:40.475687 719 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:06:40.475728 719 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:06:40.475746 719 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 04:06:40.475790 719 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.32.1
I20260810 04:06:40.477268 719 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.32.20:41905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786334794676589-31872-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786334794676589-31872-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.1.keytab
--principal=kudu/127.31.32.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.32.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786334794676589-31872-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.32.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.31.32.62:38035
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.719
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.32.1
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786334794676589-31872-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:06:40.477521 719 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:06:40.477756 719 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:06:40.482643 726 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:06:40.482708 725 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:06:40.482735 719 server_base.cc:1034] running on GCE node
W20260810 04:06:40.482662 729 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:06:40.483081 719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:06:40.483317 719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:06:40.484472 719 hybrid_clock.cc:648] HybridClock initialized: now 1786334800484456 us; error 45 us; skew 500 ppm
Aug 10 04:06:40 dist-test-slave-8w9v krb5kdc[589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334800, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:06:40.486877 719 init.cc:377] Logged in from keytab as kudu/127.31.32.1@KRBTEST.COM (short username kudu)
I20260810 04:06:40.487269 719 webserver.cc:460] Webserver started at http://127.31.32.1:43927/ using document root <none> and password file <none>
I20260810 04:06:40.487509 719 fs_manager.cc:359] Metadata directory not provided
I20260810 04:06:40.487576 719 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:06:40.487725 719 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:06:40.488798 719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786334794676589-31872-0/minicluster-data/ts-0/data/instance:
uuid: "409400b2d53242c699ea0373947ebabd"
format_stamp: "Formatted at 2026-08-10 04:06:40 on dist-test-slave-8w9v"
server_key: "df0cb780ddf4ec55726dff2ef9760a0d"
server_key_iv: "039df23db34dcef6434c2b6f629064db"
server_key_version: "encryptionkey@0"
I20260810 04:06:40.489162 719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786334794676589-31872-0/minicluster-data/ts-0/wal/instance:
uuid: "409400b2d53242c699ea0373947ebabd"
format_stamp: "Formatted at 2026-08-10 04:06:40 on dist-test-slave-8w9v"
server_key: "df0cb780ddf4ec55726dff2ef9760a0d"
server_key_iv: "039df23db34dcef6434c2b6f629064db"
server_key_version: "encryptionkey@0"
I20260810 04:06:40.490412 719 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260810 04:06:40.491099 738 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:06:40.491266 719 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 04:06:40.491369 719 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786334794676589-31872-0/minicluster-data/ts-0/data,/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786334794676589-31872-0/minicluster-data/ts-0/wal
uuid: "409400b2d53242c699ea0373947ebabd"
format_stamp: "Formatted at 2026-08-10 04:06:40 on dist-test-slave-8w9v"
server_key: "df0cb780ddf4ec55726dff2ef9760a0d"
server_key_iv: "039df23db34dcef6434c2b6f629064db"
server_key_version: "encryptionkey@0"
I20260810 04:06:40.491451 719 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786334794676589-31872-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786334794676589-31872-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786334794676589-31872-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:06:40.509927 719 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:06:40.511656 719 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:06:40.511759 719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:06:40.512238 719 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:06:40.512310 719 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:06:40.512364 719 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:06:40.512390 719 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 04:06:40 dist-test-slave-8w9v krb5kdc[589](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334800, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.1@KRBTEST.COM for kudu/127.31.32.62@KRBTEST.COM
I20260810 04:06:40.519582 719 rpc_server.cc:307] RPC server started. Bound to: 127.31.32.1:36135
I20260810 04:06:40.519685 869 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.32.1:36135 every 8 connection(s)
I20260810 04:06:40.520048 719 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786334794676589-31872-0/minicluster-data/ts-0/data/info.pb
I20260810 04:06:40.522306 696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:40.512758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.1:40439 (local address 127.31.32.62:38035)
0810 04:06:40.512828 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0810 04:06:40.512831 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:06:40.513125 (+ 294us) server_negotiation.cc:407] Connection header received
0810 04:06:40.513799 (+ 674us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:06:40.513800 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:06:40.513810 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:06:40.513858 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:06:40.514779 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:40.515379 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:40.516433 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:40.516652 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:40.516731 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:06:40.519842 (+ 3111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:06:40.519843 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:06:40.519844 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:06:40.519851 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:06:40.521230 (+ 1379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:06:40.521574 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:06:40.521574 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:06:40.521575 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:06:40.521621 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:06:40.521842 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:06:40.521843 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:06:40.521843 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:06:40.521938 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:06:40.521986 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0810 04:06:40.522181 (+ 195us) server_negotiation.cc:299] Negotiation successful
0810 04:06:40.522244 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260810 04:06:40.522466 748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:40.512740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:38035 (local address 127.31.32.1:40439)
0810 04:06:40.512949 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0810 04:06:40.512998 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0810 04:06:40.513701 (+ 703us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:06:40.513954 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:06:40.513959 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:06:40.514343 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:06:40.514676 (+ 333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:06:40.514683 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:06:40.515490 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:06:40.515492 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:06:40.516348 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:06:40.516351 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:06:40.516798 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:06:40.516798 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:06:40.516913 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:06:40.517393 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:06:40.517404 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:06:40.519713 (+ 2309us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:06:40.521305 (+ 1592us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:06:40.521311 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:06:40.521312 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:06:40.521511 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:06:40.521677 (+ 166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:06:40.521677 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:06:40.521678 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:06:40.521782 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:06:40.521991 (+ 209us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:06:40.521995 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:06:40.522096 (+ 101us) client_negotiation.cc:769] Sending connection context
0810 04:06:40.522172 (+ 76us) client_negotiation.cc:240] Negotiation successful
0810 04:06:40.522282 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":104,"threads_started":1}
I20260810 04:06:40.522957 870 heartbeater.cc:347] Connected to a master server at 127.31.32.62:38035
I20260810 04:06:40.523042 870 heartbeater.cc:464] Registering TS with master...
I20260810 04:06:40.523216 870 heartbeater.cc:511] Master 127.31.32.62:38035 requested a full tablet report, sending...
I20260810 04:06:40.523764 641 ts_manager.cc:194] Registered new tserver with Master: 409400b2d53242c699ea0373947ebabd (127.31.32.1:36135)
I20260810 04:06:40.524608 641 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.32.1@KRBTEST.COM'} at 127.31.32.1:40439
I20260810 04:06:40.529201 31872 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskws89Io/build/release/bin/kudu as pid 719
I20260810 04:06:40.529309 31872 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786334794676589-31872-0/minicluster-data/ts-0/wal/instance
I20260810 04:06:40.529497 31872 external_mini_cluster.cc:1428] Setting key f5269daaf7dec67f5847d504d35c2027
I20260810 04:06:40.529773 696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:40.524095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.1:52203 (local address 127.31.32.62:38035)
0810 04:06:40.524151 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0810 04:06:40.524153 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:06:40.524276 (+ 123us) server_negotiation.cc:407] Connection header received
0810 04:06:40.524301 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:06:40.524302 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:06:40.524311 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:06:40.524381 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:06:40.524716 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:40.525273 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:40.525984 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:40.526184 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:40.526258 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:06:40.527160 (+ 902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:06:40.527161 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:06:40.527161 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:06:40.527169 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:06:40.528581 (+ 1412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:06:40.528950 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:06:40.528951 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:06:40.528951 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:06:40.529025 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:06:40.529220 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:06:40.529220 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:06:40.529221 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:06:40.529359 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:06:40.529423 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0810 04:06:40.529598 (+ 175us) server_negotiation.cc:299] Negotiation successful
0810 04:06:40.529671 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260810 04:06:40.529776 876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:40.524036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:38035 (local address 127.31.32.1:52203)
0810 04:06:40.524203 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0810 04:06:40.524213 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 04:06:40.524268 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:06:40.524385 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:06:40.524385 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:06:40.524578 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:06:40.524651 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:06:40.524654 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:06:40.525365 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:06:40.525366 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:06:40.525879 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:06:40.525881 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:06:40.526257 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:06:40.526258 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:06:40.526334 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:06:40.526772 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:06:40.526777 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:06:40.527068 (+ 291us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:06:40.528683 (+ 1615us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:06:40.528684 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:06:40.528684 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:06:40.528874 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:06:40.529090 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:06:40.529091 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:06:40.529092 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:06:40.529149 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:06:40.529422 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:06:40.529423 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:06:40.529485 (+ 62us) client_negotiation.cc:769] Sending connection context
0810 04:06:40.529563 (+ 78us) client_negotiation.cc:240] Negotiation successful
0810 04:06:40.529670 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":71,"threads_started":1}
WARNING: no policy specified for kudu/127.31.32.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.32.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.32.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.2.keytab.
Entry for principal kudu/127.31.32.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.2.keytab.
I20260810 04:06:40.555455 31872 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskws89Io/build/release/bin/kudu
/tmp/dist-test-taskws89Io/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786334794676589-31872-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786334794676589-31872-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786334794676589-31872-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786334794676589-31872-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.32.2:0
--local_ip_for_outbound_sockets=127.31.32.2
--webserver_interface=127.31.32.2
--webserver_port=0
--tserver_master_addrs=127.31.32.62:38035
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.2.keytab
--principal=kudu/127.31.32.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.32.20:41905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786334794676589-31872-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786334794676589-31872-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786334794676589-31872-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:06:40.635700 879 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:06:40.635878 879 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:06:40.635900 879 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:06:40.636976 879 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:06:40.637012 879 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:06:40.637032 879 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 04:06:40.637049 879 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.32.2
I20260810 04:06:40.638339 879 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.32.20:41905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786334794676589-31872-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786334794676589-31872-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.2.keytab
--principal=kudu/127.31.32.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.32.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786334794676589-31872-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.32.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.31.32.62:38035
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.879
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.32.2
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786334794676589-31872-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:06:40.638548 879 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:06:40.638712 879 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:06:40.643532 889 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:06:40.643543 884 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:06:40.643589 879 server_base.cc:1034] running on GCE node
W20260810 04:06:40.643538 885 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:06:40.643896 879 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:06:40.644176 879 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:06:40.645390 879 hybrid_clock.cc:648] HybridClock initialized: now 1786334800645324 us; error 78 us; skew 500 ppm
Aug 10 04:06:40 dist-test-slave-8w9v krb5kdc[589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334800, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:06:40.647939 879 init.cc:377] Logged in from keytab as kudu/127.31.32.2@KRBTEST.COM (short username kudu)
I20260810 04:06:40.648440 879 webserver.cc:460] Webserver started at http://127.31.32.2:35801/ using document root <none> and password file <none>
I20260810 04:06:40.648703 879 fs_manager.cc:359] Metadata directory not provided
I20260810 04:06:40.648777 879 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:06:40.648924 879 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:06:40.649971 879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786334794676589-31872-0/minicluster-data/ts-1/data/instance:
uuid: "0587f4699f2640d1aa4b3035bcb0da9b"
format_stamp: "Formatted at 2026-08-10 04:06:40 on dist-test-slave-8w9v"
server_key: "6a779d1aee38e8686ea084fa8f53c54f"
server_key_iv: "bd75622a1842d17c8a059170df06c7e4"
server_key_version: "encryptionkey@0"
I20260810 04:06:40.650314 879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786334794676589-31872-0/minicluster-data/ts-1/wal/instance:
uuid: "0587f4699f2640d1aa4b3035bcb0da9b"
format_stamp: "Formatted at 2026-08-10 04:06:40 on dist-test-slave-8w9v"
server_key: "6a779d1aee38e8686ea084fa8f53c54f"
server_key_iv: "bd75622a1842d17c8a059170df06c7e4"
server_key_version: "encryptionkey@0"
I20260810 04:06:40.651577 879 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260810 04:06:40.652354 896 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:06:40.652539 879 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:06:40.652642 879 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786334794676589-31872-0/minicluster-data/ts-1/data,/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786334794676589-31872-0/minicluster-data/ts-1/wal
uuid: "0587f4699f2640d1aa4b3035bcb0da9b"
format_stamp: "Formatted at 2026-08-10 04:06:40 on dist-test-slave-8w9v"
server_key: "6a779d1aee38e8686ea084fa8f53c54f"
server_key_iv: "bd75622a1842d17c8a059170df06c7e4"
server_key_version: "encryptionkey@0"
I20260810 04:06:40.652720 879 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786334794676589-31872-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786334794676589-31872-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786334794676589-31872-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:06:40.660341 879 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:06:40.662099 879 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:06:40.662286 879 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:06:40.662783 879 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:06:40.662848 879 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:06:40.662885 879 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:06:40.662918 879 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 04:06:40 dist-test-slave-8w9v krb5kdc[589](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334800, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.2@KRBTEST.COM for kudu/127.31.32.62@KRBTEST.COM
I20260810 04:06:40.670886 879 rpc_server.cc:307] RPC server started. Bound to: 127.31.32.2:40031
I20260810 04:06:40.670970 1022 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.32.2:40031 every 8 connection(s)
I20260810 04:06:40.671281 879 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786334794676589-31872-0/minicluster-data/ts-1/data/info.pb
I20260810 04:06:40.672963 696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:40.663277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.2:33021 (local address 127.31.32.62:38035)
0810 04:06:40.663364 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0810 04:06:40.663366 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:06:40.663548 (+ 182us) server_negotiation.cc:407] Connection header received
0810 04:06:40.663962 (+ 414us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:06:40.663964 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:06:40.663981 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:06:40.664069 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:06:40.664702 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:40.665528 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:40.667059 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:40.667386 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:40.667461 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:06:40.670206 (+ 2745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:06:40.670207 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:06:40.670208 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:06:40.670214 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:06:40.671795 (+ 1581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:06:40.672190 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:06:40.672191 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:06:40.672192 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:06:40.672274 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:06:40.672517 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:06:40.672517 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:06:40.672517 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:06:40.672611 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:06:40.672657 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0810 04:06:40.672843 (+ 186us) server_negotiation.cc:299] Negotiation successful
0810 04:06:40.672901 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 04:06:40.673076 906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:40.663190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:38035 (local address 127.31.32.2:33021)
0810 04:06:40.663393 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0810 04:06:40.663430 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0810 04:06:40.663860 (+ 430us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:06:40.664190 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:06:40.664194 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:06:40.664433 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:06:40.664601 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:06:40.664605 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:06:40.665654 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:06:40.665655 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:06:40.666985 (+ 1330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:06:40.666988 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:06:40.667478 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:06:40.667479 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:06:40.667621 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:06:40.668196 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:06:40.668212 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:06:40.670104 (+ 1892us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:06:40.671879 (+ 1775us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:06:40.671882 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:06:40.671884 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:06:40.672078 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:06:40.672365 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:06:40.672366 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:06:40.672366 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:06:40.672461 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:06:40.672660 (+ 199us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:06:40.672663 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:06:40.672758 (+ 95us) client_negotiation.cc:769] Sending connection context
0810 04:06:40.672822 (+ 64us) client_negotiation.cc:240] Negotiation successful
0810 04:06:40.672924 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":95,"threads_started":1}
I20260810 04:06:40.673556 1023 heartbeater.cc:347] Connected to a master server at 127.31.32.62:38035
I20260810 04:06:40.673636 1023 heartbeater.cc:464] Registering TS with master...
I20260810 04:06:40.673795 1023 heartbeater.cc:511] Master 127.31.32.62:38035 requested a full tablet report, sending...
I20260810 04:06:40.674204 639 ts_manager.cc:194] Registered new tserver with Master: 0587f4699f2640d1aa4b3035bcb0da9b (127.31.32.2:40031)
I20260810 04:06:40.674702 639 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.32.2@KRBTEST.COM'} at 127.31.32.2:33021
I20260810 04:06:40.679585 1029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:40.674484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:38035 (local address 127.31.32.2:48277)
0810 04:06:40.674619 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0810 04:06:40.674629 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 04:06:40.674686 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:06:40.674902 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:06:40.674903 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:06:40.675122 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:06:40.675212 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:06:40.675214 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:06:40.675822 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:06:40.675823 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:06:40.676378 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:06:40.676380 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:06:40.676688 (+ 308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:06:40.676689 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:06:40.676742 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:06:40.677133 (+ 391us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:06:40.677138 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:06:40.677427 (+ 289us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:06:40.678660 (+ 1233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:06:40.678661 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:06:40.678662 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:06:40.678809 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:06:40.679014 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:06:40.679014 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:06:40.679015 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:06:40.679099 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:06:40.679356 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:06:40.679357 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:06:40.679402 (+ 45us) client_negotiation.cc:769] Sending connection context
0810 04:06:40.679455 (+ 53us) client_negotiation.cc:240] Negotiation successful
0810 04:06:40.679525 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":62,"threads_started":1}
I20260810 04:06:40.679600 696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:40.674530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.2:48277 (local address 127.31.32.62:38035)
0810 04:06:40.674565 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0810 04:06:40.674566 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 04:06:40.674698 (+ 132us) server_negotiation.cc:407] Connection header received
0810 04:06:40.674726 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:06:40.674727 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:06:40.674738 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:06:40.674813 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:06:40.675294 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:40.675726 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:40.676445 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:40.676621 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:40.676680 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:06:40.677501 (+ 821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:06:40.677502 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:06:40.677503 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:06:40.677508 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:06:40.678567 (+ 1059us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:06:40.678894 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:06:40.678895 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:06:40.678895 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:06:40.678948 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:06:40.679164 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:06:40.679164 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:06:40.679165 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:06:40.679291 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:06:40.679344 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0810 04:06:40.679486 (+ 142us) server_negotiation.cc:299] Negotiation successful
0810 04:06:40.679546 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260810 04:06:40.680179 31872 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskws89Io/build/release/bin/kudu as pid 879
I20260810 04:06:40.680274 31872 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786334794676589-31872-0/minicluster-data/ts-1/wal/instance
I20260810 04:06:40.680514 31872 external_mini_cluster.cc:1428] Setting key 405db730c412c242448aaed0a579ef65
WARNING: no policy specified for kudu/127.31.32.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.32.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.32.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.3.keytab.
Entry for principal kudu/127.31.32.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.3.keytab.
I20260810 04:06:40.705080 31872 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskws89Io/build/release/bin/kudu
/tmp/dist-test-taskws89Io/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786334794676589-31872-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786334794676589-31872-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786334794676589-31872-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786334794676589-31872-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.32.3:0
--local_ip_for_outbound_sockets=127.31.32.3
--webserver_interface=127.31.32.3
--webserver_port=0
--tserver_master_addrs=127.31.32.62:38035
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.3.keytab
--principal=kudu/127.31.32.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.32.20:41905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786334794676589-31872-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786334794676589-31872-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786334794676589-31872-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:06:40.782560 1034 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:06:40.782743 1034 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:06:40.782764 1034 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:06:40.783862 1034 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:06:40.783900 1034 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:06:40.783919 1034 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 04:06:40.783959 1034 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.32.3
I20260810 04:06:40.785360 1034 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.32.20:41905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786334794676589-31872-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786334794676589-31872-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.3.keytab
--principal=kudu/127.31.32.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.32.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786334794676589-31872-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.32.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.31.32.62:38035
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1034
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.32.3
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786334794676589-31872-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:06:40.785591 1034 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:06:40.785804 1034 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:06:40.790556 1039 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:06:40.790580 1042 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:06:40.790567 1040 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:06:40.790820 1034 server_base.cc:1034] running on GCE node
I20260810 04:06:40.791028 1034 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:06:40.791265 1034 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:06:40.792428 1034 hybrid_clock.cc:648] HybridClock initialized: now 1786334800792409 us; error 30 us; skew 500 ppm
Aug 10 04:06:40 dist-test-slave-8w9v krb5kdc[589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334800, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:06:40.794852 1034 init.cc:377] Logged in from keytab as kudu/127.31.32.3@KRBTEST.COM (short username kudu)
I20260810 04:06:40.795287 1034 webserver.cc:460] Webserver started at http://127.31.32.3:41963/ using document root <none> and password file <none>
I20260810 04:06:40.795481 1034 fs_manager.cc:359] Metadata directory not provided
I20260810 04:06:40.795521 1034 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:06:40.795600 1034 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:06:40.796579 1034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786334794676589-31872-0/minicluster-data/ts-2/data/instance:
uuid: "f12f746012f14fa983ea93b29069958e"
format_stamp: "Formatted at 2026-08-10 04:06:40 on dist-test-slave-8w9v"
server_key: "a5fb1386474954704ab30be4161d9aa6"
server_key_iv: "478504a21fd5c0762c844e1755b82062"
server_key_version: "encryptionkey@0"
I20260810 04:06:40.796891 1034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786334794676589-31872-0/minicluster-data/ts-2/wal/instance:
uuid: "f12f746012f14fa983ea93b29069958e"
format_stamp: "Formatted at 2026-08-10 04:06:40 on dist-test-slave-8w9v"
server_key: "a5fb1386474954704ab30be4161d9aa6"
server_key_iv: "478504a21fd5c0762c844e1755b82062"
server_key_version: "encryptionkey@0"
I20260810 04:06:40.798163 1034 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 04:06:40.798789 1049 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:06:40.798969 1034 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 04:06:40.799070 1034 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786334794676589-31872-0/minicluster-data/ts-2/data,/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786334794676589-31872-0/minicluster-data/ts-2/wal
uuid: "f12f746012f14fa983ea93b29069958e"
format_stamp: "Formatted at 2026-08-10 04:06:40 on dist-test-slave-8w9v"
server_key: "a5fb1386474954704ab30be4161d9aa6"
server_key_iv: "478504a21fd5c0762c844e1755b82062"
server_key_version: "encryptionkey@0"
I20260810 04:06:40.799135 1034 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786334794676589-31872-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786334794676589-31872-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786334794676589-31872-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:06:40.833213 1034 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:06:40.835366 1034 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:06:40.835541 1034 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:06:40.836069 1034 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:06:40.836112 1034 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:06:40.836197 1034 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:06:40.836228 1034 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:06:40.843029 1034 rpc_server.cc:307] RPC server started. Bound to: 127.31.32.3:45985
I20260810 04:06:40.843150 1179 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.32.3:45985 every 8 connection(s)
Aug 10 04:06:40 dist-test-slave-8w9v krb5kdc[589](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334800, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.3@KRBTEST.COM for kudu/127.31.32.62@KRBTEST.COM
I20260810 04:06:40.843452 1034 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786334794676589-31872-0/minicluster-data/ts-2/data/info.pb
I20260810 04:06:40.846709 696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:40.836646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.3:54441 (local address 127.31.32.62:38035)
0810 04:06:40.836720 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0810 04:06:40.836722 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:06:40.836940 (+ 218us) server_negotiation.cc:407] Connection header received
0810 04:06:40.837415 (+ 475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:06:40.837416 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:06:40.837429 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:06:40.837513 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:06:40.838397 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:40.839250 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:40.840710 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:40.841028 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:40.841088 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:06:40.843844 (+ 2756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:06:40.843845 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:06:40.843845 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:06:40.843854 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:06:40.845229 (+ 1375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:06:40.845638 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:06:40.845639 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:06:40.845639 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:06:40.845699 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:06:40.846043 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:06:40.846044 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:06:40.846045 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:06:40.846167 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:06:40.846241 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0810 04:06:40.846516 (+ 275us) server_negotiation.cc:299] Negotiation successful
0810 04:06:40.846618 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260810 04:06:40.846875 1061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:40.836615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:38035 (local address 127.31.32.3:54441)
0810 04:06:40.836811 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0810 04:06:40.836850 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0810 04:06:40.837319 (+ 469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:06:40.837590 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:06:40.837593 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:06:40.837987 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:06:40.838252 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:06:40.838260 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:06:40.839407 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:06:40.839408 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:06:40.840602 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:06:40.840605 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:06:40.841095 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:06:40.841096 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:06:40.841233 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:06:40.841810 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:06:40.841820 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:06:40.843716 (+ 1896us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:06:40.845333 (+ 1617us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:06:40.845339 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:06:40.845340 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:06:40.845547 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:06:40.845764 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:06:40.845764 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:06:40.845765 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:06:40.845923 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:06:40.846232 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:06:40.846239 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:06:40.846385 (+ 146us) client_negotiation.cc:769] Sending connection context
0810 04:06:40.846484 (+ 99us) client_negotiation.cc:240] Negotiation successful
0810 04:06:40.846649 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":128,"threads_started":1}
I20260810 04:06:40.847330 1180 heartbeater.cc:347] Connected to a master server at 127.31.32.62:38035
I20260810 04:06:40.847420 1180 heartbeater.cc:464] Registering TS with master...
I20260810 04:06:40.847608 1180 heartbeater.cc:511] Master 127.31.32.62:38035 requested a full tablet report, sending...
I20260810 04:06:40.847998 641 ts_manager.cc:194] Registered new tserver with Master: f12f746012f14fa983ea93b29069958e (127.31.32.3:45985)
I20260810 04:06:40.848547 641 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.32.3@KRBTEST.COM'} at 127.31.32.3:54441
I20260810 04:06:40.849604 31872 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskws89Io/build/release/bin/kudu as pid 1034
I20260810 04:06:40.849674 31872 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786334794676589-31872-0/minicluster-data/ts-2/wal/instance
I20260810 04:06:40.849851 31872 external_mini_cluster.cc:1428] Setting key 8fd139ac6d637e5a609921ce3c37b08c
I20260810 04:06:40.850958 31872 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 04:06:40.853896 696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:40.848329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.3:43433 (local address 127.31.32.62:38035)
0810 04:06:40.848361 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0810 04:06:40.848362 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 04:06:40.848495 (+ 133us) server_negotiation.cc:407] Connection header received
0810 04:06:40.848519 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:06:40.848520 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:06:40.848529 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:06:40.848591 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:06:40.848923 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:40.849319 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:40.850100 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:40.850378 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:40.850451 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:06:40.851267 (+ 816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:06:40.851268 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:06:40.851269 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:06:40.851276 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:06:40.852786 (+ 1510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:06:40.853099 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:06:40.853099 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:06:40.853100 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:06:40.853158 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:06:40.853381 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:06:40.853381 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:06:40.853382 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:06:40.853495 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:06:40.853527 (+ 32us) server_negotiation.cc:1035] Waiting for connection context
0810 04:06:40.853716 (+ 189us) server_negotiation.cc:299] Negotiation successful
0810 04:06:40.853809 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260810 04:06:40.853914 1188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:40.848314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:38035 (local address 127.31.32.3:43433)
0810 04:06:40.848427 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0810 04:06:40.848435 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0810 04:06:40.848484 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:06:40.848592 (+ 108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:06:40.848592 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:06:40.848789 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:06:40.848866 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:06:40.848868 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:06:40.849419 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:06:40.849420 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:06:40.850009 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:06:40.850011 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:06:40.850434 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:06:40.850435 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:06:40.850510 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:06:40.850868 (+ 358us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:06:40.850872 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:06:40.851175 (+ 303us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:06:40.852865 (+ 1690us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:06:40.852866 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:06:40.852866 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:06:40.853020 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:06:40.853239 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:06:40.853240 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:06:40.853241 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:06:40.853309 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:06:40.853564 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:06:40.853565 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:06:40.853598 (+ 33us) client_negotiation.cc:769] Sending connection context
0810 04:06:40.853672 (+ 74us) client_negotiation.cc:240] Negotiation successful
0810 04:06:40.853811 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":63,"threads_started":1}
Aug 10 04:06:40 dist-test-slave-8w9v krb5kdc[589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334800, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 10 04:06:40 dist-test-slave-8w9v krb5kdc[589](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334800, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.31.32.62@KRBTEST.COM
I20260810 04:06:40.875543 696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:40.868459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33394 (local address 127.31.32.62:38035)
0810 04:06:40.868498 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0810 04:06:40.868500 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:06:40.868507 (+ 7us) server_negotiation.cc:407] Connection header received
0810 04:06:40.868527 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:06:40.868528 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:06:40.868536 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:06:40.868603 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:06:40.869163 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:40.869614 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:40.870344 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:40.870524 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:40.870597 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:06:40.872918 (+ 2321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:06:40.872919 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:06:40.872920 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:06:40.872927 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:06:40.874375 (+ 1448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:06:40.874715 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:06:40.874716 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:06:40.874716 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:06:40.874790 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:06:40.875046 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:06:40.875047 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:06:40.875047 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:06:40.875155 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:06:40.875250 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0810 04:06:40.875356 (+ 106us) server_negotiation.cc:299] Negotiation successful
0810 04:06:40.875430 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260810 04:06:40.877269 641 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33394:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 04:06:40.878773 641 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 04:06:40.885293 1202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:40.881282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.1:36135 (local address 127.0.0.1:35874)
0810 04:06:40.881715 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0810 04:06:40.881748 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0810 04:06:40.881823 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:06:40.882056 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:06:40.882057 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:06:40.882058 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:06:40.882116 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:06:40.882118 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:06:40.882936 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:06:40.882936 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:06:40.883987 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:06:40.883990 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:06:40.884978 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:06:40.884979 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:06:40.885062 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:06:40.885075 (+ 13us) client_negotiation.cc:769] Sending connection context
0810 04:06:40.885150 (+ 75us) client_negotiation.cc:240] Negotiation successful
0810 04:06:40.885198 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":413,"mutex_wait_us":4,"thread_start_us":91,"threads_started":1}
I20260810 04:06:40.885330 1201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:40.881420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35874 (local address 127.31.32.1:36135)
0810 04:06:40.881633 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0810 04:06:40.881638 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 04:06:40.881825 (+ 187us) server_negotiation.cc:407] Connection header received
0810 04:06:40.881888 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:06:40.881890 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:06:40.881907 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:06:40.881982 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:06:40.882205 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:40.882858 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:40.884074 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:40.884818 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:40.884889 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:06:40.884943 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0810 04:06:40.885181 (+ 238us) server_negotiation.cc:299] Negotiation successful
0810 04:06:40.885229 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":95,"threads_started":1}
I20260810 04:06:40.885572 1199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:40.881282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.2:40031 (local address 127.0.0.1:41090)
0810 04:06:40.881424 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0810 04:06:40.881460 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0810 04:06:40.881521 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:06:40.882002 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:06:40.882005 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:06:40.882007 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:06:40.882090 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:06:40.882092 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:06:40.883371 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:06:40.883372 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:06:40.884408 (+ 1036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:06:40.884410 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:06:40.885325 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:06:40.885326 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:06:40.885402 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:06:40.885406 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 04:06:40.885440 (+ 34us) client_negotiation.cc:240] Negotiation successful
0810 04:06:40.885480 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"mutex_wait_us":4,"thread_start_us":43,"threads_started":1}
I20260810 04:06:40.885969 1205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:40.881623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41090 (local address 127.31.32.2:40031)
0810 04:06:40.881817 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0810 04:06:40.881821 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 04:06:40.881831 (+ 10us) server_negotiation.cc:407] Connection header received
0810 04:06:40.881871 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:06:40.881873 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:06:40.881889 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:06:40.882171 (+ 282us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:06:40.882237 (+ 66us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:40.883273 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:40.884514 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:40.885218 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:40.885745 (+ 527us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:06:40.885814 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0810 04:06:40.885838 (+ 24us) server_negotiation.cc:299] Negotiation successful
0810 04:06:40.885880 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":101,"threads_started":1}
I20260810 04:06:40.886179 1200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:40.881470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.3:45985 (local address 127.0.0.1:35194)
0810 04:06:40.881817 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0810 04:06:40.881829 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 04:06:40.881885 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:06:40.882111 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:06:40.882112 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:06:40.882113 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:06:40.882151 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:06:40.882153 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:06:40.884158 (+ 2005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:06:40.884159 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:06:40.884968 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:06:40.884970 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:06:40.885954 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:06:40.885955 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:06:40.886019 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:06:40.886022 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 04:06:40.886070 (+ 48us) client_negotiation.cc:240] Negotiation successful
0810 04:06:40.886108 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":72,"threads_started":1}
I20260810 04:06:40.886162 795 tablet_service.cc:1467] Processing CreateTablet for tablet f782fb179989449e8e992ced2e7ae69a (DEFAULT_TABLE table=test-table [id=d02ba4a0cb644823a8f76fd57eab7361]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:06:40.886461 1203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:40.881558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35194 (local address 127.31.32.3:45985)
0810 04:06:40.881740 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0810 04:06:40.881744 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 04:06:40.881942 (+ 198us) server_negotiation.cc:407] Connection header received
0810 04:06:40.881979 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:06:40.881981 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:06:40.881994 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:06:40.882056 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:06:40.882225 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:40.883288 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:40.885055 (+ 1767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:40.885802 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:40.885869 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:06:40.885921 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0810 04:06:40.886316 (+ 395us) server_negotiation.cc:299] Negotiation successful
0810 04:06:40.886370 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":83,"threads_started":1}
I20260810 04:06:40.886534 795 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f782fb179989449e8e992ced2e7ae69a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:06:40.887140 1109 tablet_service.cc:1467] Processing CreateTablet for tablet f782fb179989449e8e992ced2e7ae69a (DEFAULT_TABLE table=test-table [id=d02ba4a0cb644823a8f76fd57eab7361]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:06:40.887476 1109 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f782fb179989449e8e992ced2e7ae69a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:06:40.890246 1206 tablet_bootstrap.cc:492] T f782fb179989449e8e992ced2e7ae69a P 409400b2d53242c699ea0373947ebabd: Bootstrap starting.
I20260810 04:06:40.890578 949 tablet_service.cc:1467] Processing CreateTablet for tablet f782fb179989449e8e992ced2e7ae69a (DEFAULT_TABLE table=test-table [id=d02ba4a0cb644823a8f76fd57eab7361]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:06:40.890699 1207 tablet_bootstrap.cc:492] T f782fb179989449e8e992ced2e7ae69a P f12f746012f14fa983ea93b29069958e: Bootstrap starting.
I20260810 04:06:40.890914 949 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f782fb179989449e8e992ced2e7ae69a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:06:40.891067 1206 tablet_bootstrap.cc:654] T f782fb179989449e8e992ced2e7ae69a P 409400b2d53242c699ea0373947ebabd: Neither blocks nor log segments found. Creating new log.
I20260810 04:06:40.891438 1206 log.cc:824] T f782fb179989449e8e992ced2e7ae69a P 409400b2d53242c699ea0373947ebabd: Log is configured to *not* fsync() on all Append() calls
I20260810 04:06:40.891636 1207 tablet_bootstrap.cc:654] T f782fb179989449e8e992ced2e7ae69a P f12f746012f14fa983ea93b29069958e: Neither blocks nor log segments found. Creating new log.
I20260810 04:06:40.891991 1207 log.cc:824] T f782fb179989449e8e992ced2e7ae69a P f12f746012f14fa983ea93b29069958e: Log is configured to *not* fsync() on all Append() calls
I20260810 04:06:40.892274 1206 tablet_bootstrap.cc:492] T f782fb179989449e8e992ced2e7ae69a P 409400b2d53242c699ea0373947ebabd: No bootstrap required, opened a new log
I20260810 04:06:40.892347 1206 ts_tablet_manager.cc:1397] T f782fb179989449e8e992ced2e7ae69a P 409400b2d53242c699ea0373947ebabd: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 04:06:40.892817 1207 tablet_bootstrap.cc:492] T f782fb179989449e8e992ced2e7ae69a P f12f746012f14fa983ea93b29069958e: No bootstrap required, opened a new log
I20260810 04:06:40.892885 1207 ts_tablet_manager.cc:1397] T f782fb179989449e8e992ced2e7ae69a P f12f746012f14fa983ea93b29069958e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 04:06:40.893694 1206 raft_consensus.cc:348] T f782fb179989449e8e992ced2e7ae69a P 409400b2d53242c699ea0373947ebabd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0587f4699f2640d1aa4b3035bcb0da9b" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 40031 } } peers { permanent_uuid: "409400b2d53242c699ea0373947ebabd" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 36135 } } peers { permanent_uuid: "f12f746012f14fa983ea93b29069958e" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 45985 } }
I20260810 04:06:40.893846 1206 raft_consensus.cc:374] T f782fb179989449e8e992ced2e7ae69a P 409400b2d53242c699ea0373947ebabd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:06:40.893888 1206 raft_consensus.cc:729] T f782fb179989449e8e992ced2e7ae69a P 409400b2d53242c699ea0373947ebabd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 409400b2d53242c699ea0373947ebabd, State: Initialized, Role: FOLLOWER
I20260810 04:06:40.894016 1206 consensus_queue.cc:260] T f782fb179989449e8e992ced2e7ae69a P 409400b2d53242c699ea0373947ebabd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0587f4699f2640d1aa4b3035bcb0da9b" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 40031 } } peers { permanent_uuid: "409400b2d53242c699ea0373947ebabd" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 36135 } } peers { permanent_uuid: "f12f746012f14fa983ea93b29069958e" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 45985 } }
I20260810 04:06:40.894271 1206 ts_tablet_manager.cc:1428] T f782fb179989449e8e992ced2e7ae69a P 409400b2d53242c699ea0373947ebabd: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260810 04:06:40.894388 870 heartbeater.cc:503] Master 127.31.32.62:38035 was elected leader, sending a full tablet report...
I20260810 04:06:40.894651 1211 tablet_bootstrap.cc:492] T f782fb179989449e8e992ced2e7ae69a P 0587f4699f2640d1aa4b3035bcb0da9b: Bootstrap starting.
I20260810 04:06:40.894821 1207 raft_consensus.cc:348] T f782fb179989449e8e992ced2e7ae69a P f12f746012f14fa983ea93b29069958e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0587f4699f2640d1aa4b3035bcb0da9b" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 40031 } } peers { permanent_uuid: "409400b2d53242c699ea0373947ebabd" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 36135 } } peers { permanent_uuid: "f12f746012f14fa983ea93b29069958e" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 45985 } }
I20260810 04:06:40.894965 1207 raft_consensus.cc:374] T f782fb179989449e8e992ced2e7ae69a P f12f746012f14fa983ea93b29069958e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:06:40.895002 1207 raft_consensus.cc:729] T f782fb179989449e8e992ced2e7ae69a P f12f746012f14fa983ea93b29069958e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f12f746012f14fa983ea93b29069958e, State: Initialized, Role: FOLLOWER
I20260810 04:06:40.895114 1207 consensus_queue.cc:260] T f782fb179989449e8e992ced2e7ae69a P f12f746012f14fa983ea93b29069958e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0587f4699f2640d1aa4b3035bcb0da9b" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 40031 } } peers { permanent_uuid: "409400b2d53242c699ea0373947ebabd" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 36135 } } peers { permanent_uuid: "f12f746012f14fa983ea93b29069958e" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 45985 } }
I20260810 04:06:40.895347 1211 tablet_bootstrap.cc:654] T f782fb179989449e8e992ced2e7ae69a P 0587f4699f2640d1aa4b3035bcb0da9b: Neither blocks nor log segments found. Creating new log.
I20260810 04:06:40.895376 1207 ts_tablet_manager.cc:1428] T f782fb179989449e8e992ced2e7ae69a P f12f746012f14fa983ea93b29069958e: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260810 04:06:40.895493 1180 heartbeater.cc:503] Master 127.31.32.62:38035 was elected leader, sending a full tablet report...
I20260810 04:06:40.895679 1211 log.cc:824] T f782fb179989449e8e992ced2e7ae69a P 0587f4699f2640d1aa4b3035bcb0da9b: Log is configured to *not* fsync() on all Append() calls
I20260810 04:06:40.896540 1211 tablet_bootstrap.cc:492] T f782fb179989449e8e992ced2e7ae69a P 0587f4699f2640d1aa4b3035bcb0da9b: No bootstrap required, opened a new log
I20260810 04:06:40.896632 1211 ts_tablet_manager.cc:1397] T f782fb179989449e8e992ced2e7ae69a P 0587f4699f2640d1aa4b3035bcb0da9b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 04:06:40.897984 1211 raft_consensus.cc:348] T f782fb179989449e8e992ced2e7ae69a P 0587f4699f2640d1aa4b3035bcb0da9b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0587f4699f2640d1aa4b3035bcb0da9b" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 40031 } } peers { permanent_uuid: "409400b2d53242c699ea0373947ebabd" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 36135 } } peers { permanent_uuid: "f12f746012f14fa983ea93b29069958e" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 45985 } }
I20260810 04:06:40.898119 1211 raft_consensus.cc:374] T f782fb179989449e8e992ced2e7ae69a P 0587f4699f2640d1aa4b3035bcb0da9b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:06:40.898164 1211 raft_consensus.cc:729] T f782fb179989449e8e992ced2e7ae69a P 0587f4699f2640d1aa4b3035bcb0da9b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0587f4699f2640d1aa4b3035bcb0da9b, State: Initialized, Role: FOLLOWER
I20260810 04:06:40.898268 1211 consensus_queue.cc:260] T f782fb179989449e8e992ced2e7ae69a P 0587f4699f2640d1aa4b3035bcb0da9b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0587f4699f2640d1aa4b3035bcb0da9b" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 40031 } } peers { permanent_uuid: "409400b2d53242c699ea0373947ebabd" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 36135 } } peers { permanent_uuid: "f12f746012f14fa983ea93b29069958e" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 45985 } }
I20260810 04:06:40.898517 1211 ts_tablet_manager.cc:1428] T f782fb179989449e8e992ced2e7ae69a P 0587f4699f2640d1aa4b3035bcb0da9b: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260810 04:06:40.898566 1023 heartbeater.cc:503] Master 127.31.32.62:38035 was elected leader, sending a full tablet report...
I20260810 04:06:40.909525 1212 raft_consensus.cc:482] T f782fb179989449e8e992ced2e7ae69a P f12f746012f14fa983ea93b29069958e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 04:06:40.909603 1212 raft_consensus.cc:504] T f782fb179989449e8e992ced2e7ae69a P f12f746012f14fa983ea93b29069958e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0587f4699f2640d1aa4b3035bcb0da9b" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 40031 } } peers { permanent_uuid: "409400b2d53242c699ea0373947ebabd" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 36135 } } peers { permanent_uuid: "f12f746012f14fa983ea93b29069958e" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 45985 } }
I20260810 04:06:40.909932 1212 leader_election.cc:288] T f782fb179989449e8e992ced2e7ae69a P f12f746012f14fa983ea93b29069958e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0587f4699f2640d1aa4b3035bcb0da9b (127.31.32.2:40031), 409400b2d53242c699ea0373947ebabd (127.31.32.1:36135)
I20260810 04:06:40.913561 1201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:40.910148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.3:46365 (local address 127.31.32.1:36135)
0810 04:06:40.910188 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0810 04:06:40.910190 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:06:40.910349 (+ 159us) server_negotiation.cc:407] Connection header received
0810 04:06:40.910410 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:06:40.910411 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:06:40.910420 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:06:40.910472 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:06:40.910618 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:40.911257 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:40.912692 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:40.913213 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:40.913287 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:06:40.913297 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0810 04:06:40.913462 (+ 165us) server_negotiation.cc:299] Negotiation successful
0810 04:06:40.913496 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260810 04:06:40.913561 1205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:40.910178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.3:48969 (local address 127.31.32.2:40031)
0810 04:06:40.910230 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0810 04:06:40.910232 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:06:40.910239 (+ 7us) server_negotiation.cc:407] Connection header received
0810 04:06:40.910264 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:06:40.910270 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:06:40.910278 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:06:40.910366 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:06:40.910530 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:40.911257 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:40.912692 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:40.913213 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:40.913288 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:06:40.913297 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0810 04:06:40.913462 (+ 165us) server_negotiation.cc:299] Negotiation successful
0810 04:06:40.913496 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260810 04:06:40.913575 1061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:40.910069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.2:40031 (local address 127.31.32.3:48969)
0810 04:06:40.910124 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0810 04:06:40.910136 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 04:06:40.910179 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:06:40.910372 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:06:40.910374 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:06:40.910375 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:06:40.910445 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:06:40.910462 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:06:40.911335 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:06:40.911336 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:06:40.912604 (+ 1268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:06:40.912607 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:06:40.913296 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:06:40.913297 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:06:40.913381 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:06:40.913385 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 04:06:40.913442 (+ 57us) client_negotiation.cc:240] Negotiation successful
0810 04:06:40.913489 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"mutex_wait_us":59}
I20260810 04:06:40.913580 1216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:40.910111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.1:36135 (local address 127.31.32.3:46365)
0810 04:06:40.910274 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0810 04:06:40.910281 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0810 04:06:40.910360 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:06:40.910484 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:06:40.910484 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:06:40.910486 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:06:40.910549 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:06:40.910550 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:06:40.911345 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:06:40.911345 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:06:40.912604 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:06:40.912607 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:06:40.913297 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:06:40.913297 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:06:40.913390 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:06:40.913393 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 04:06:40.913442 (+ 49us) client_negotiation.cc:240] Negotiation successful
0810 04:06:40.913489 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"mutex_wait_us":70,"thread_start_us":88,"threads_started":1}
I20260810 04:06:40.913954 973 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f782fb179989449e8e992ced2e7ae69a" candidate_uuid: "f12f746012f14fa983ea93b29069958e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0587f4699f2640d1aa4b3035bcb0da9b" is_pre_election: true
I20260810 04:06:40.914083 973 raft_consensus.cc:2454] T f782fb179989449e8e992ced2e7ae69a P 0587f4699f2640d1aa4b3035bcb0da9b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12f746012f14fa983ea93b29069958e in term 0.
I20260810 04:06:40.914258 1050 leader_election.cc:302] T f782fb179989449e8e992ced2e7ae69a P f12f746012f14fa983ea93b29069958e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0587f4699f2640d1aa4b3035bcb0da9b, f12f746012f14fa983ea93b29069958e; no voters:
I20260810 04:06:40.914265 818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f782fb179989449e8e992ced2e7ae69a" candidate_uuid: "f12f746012f14fa983ea93b29069958e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "409400b2d53242c699ea0373947ebabd" is_pre_election: true
I20260810 04:06:40.914367 818 raft_consensus.cc:2454] T f782fb179989449e8e992ced2e7ae69a P 409400b2d53242c699ea0373947ebabd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12f746012f14fa983ea93b29069958e in term 0.
I20260810 04:06:40.914511 1212 raft_consensus.cc:2781] T f782fb179989449e8e992ced2e7ae69a P f12f746012f14fa983ea93b29069958e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 04:06:40.914565 1212 raft_consensus.cc:482] T f782fb179989449e8e992ced2e7ae69a P f12f746012f14fa983ea93b29069958e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 04:06:40.914587 1212 raft_consensus.cc:3037] T f782fb179989449e8e992ced2e7ae69a P f12f746012f14fa983ea93b29069958e [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:06:40.915299 1212 raft_consensus.cc:504] T f782fb179989449e8e992ced2e7ae69a P f12f746012f14fa983ea93b29069958e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0587f4699f2640d1aa4b3035bcb0da9b" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 40031 } } peers { permanent_uuid: "409400b2d53242c699ea0373947ebabd" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 36135 } } peers { permanent_uuid: "f12f746012f14fa983ea93b29069958e" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 45985 } }
I20260810 04:06:40.915444 1212 leader_election.cc:288] T f782fb179989449e8e992ced2e7ae69a P f12f746012f14fa983ea93b29069958e [CANDIDATE]: Term 1 election: Requested vote from peers 0587f4699f2640d1aa4b3035bcb0da9b (127.31.32.2:40031), 409400b2d53242c699ea0373947ebabd (127.31.32.1:36135)
I20260810 04:06:40.915601 818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f782fb179989449e8e992ced2e7ae69a" candidate_uuid: "f12f746012f14fa983ea93b29069958e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "409400b2d53242c699ea0373947ebabd"
I20260810 04:06:40.915697 818 raft_consensus.cc:3037] T f782fb179989449e8e992ced2e7ae69a P 409400b2d53242c699ea0373947ebabd [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:06:40.915719 973 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f782fb179989449e8e992ced2e7ae69a" candidate_uuid: "f12f746012f14fa983ea93b29069958e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0587f4699f2640d1aa4b3035bcb0da9b"
I20260810 04:06:40.915812 973 raft_consensus.cc:3037] T f782fb179989449e8e992ced2e7ae69a P 0587f4699f2640d1aa4b3035bcb0da9b [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:06:40.916646 818 raft_consensus.cc:2454] T f782fb179989449e8e992ced2e7ae69a P 409400b2d53242c699ea0373947ebabd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f12f746012f14fa983ea93b29069958e in term 1.
I20260810 04:06:40.916678 973 raft_consensus.cc:2454] T f782fb179989449e8e992ced2e7ae69a P 0587f4699f2640d1aa4b3035bcb0da9b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f12f746012f14fa983ea93b29069958e in term 1.
I20260810 04:06:40.917271 1053 leader_election.cc:302] T f782fb179989449e8e992ced2e7ae69a P f12f746012f14fa983ea93b29069958e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 409400b2d53242c699ea0373947ebabd, f12f746012f14fa983ea93b29069958e; no voters:
I20260810 04:06:40.917443 1212 raft_consensus.cc:2781] T f782fb179989449e8e992ced2e7ae69a P f12f746012f14fa983ea93b29069958e [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:06:40.917613 1212 raft_consensus.cc:686] T f782fb179989449e8e992ced2e7ae69a P f12f746012f14fa983ea93b29069958e [term 1 LEADER]: Becoming Leader. State: Replica: f12f746012f14fa983ea93b29069958e, State: Running, Role: LEADER
I20260810 04:06:40.917716 1212 consensus_queue.cc:237] T f782fb179989449e8e992ced2e7ae69a P f12f746012f14fa983ea93b29069958e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0587f4699f2640d1aa4b3035bcb0da9b" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 40031 } } peers { permanent_uuid: "409400b2d53242c699ea0373947ebabd" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 36135 } } peers { permanent_uuid: "f12f746012f14fa983ea93b29069958e" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 45985 } }
I20260810 04:06:40.918495 641 catalog_manager.cc:5495] T f782fb179989449e8e992ced2e7ae69a P f12f746012f14fa983ea93b29069958e reported cstate change: term changed from 0 to 1, leader changed from <none> to f12f746012f14fa983ea93b29069958e (127.31.32.3). New cstate: current_term: 1 leader_uuid: "f12f746012f14fa983ea93b29069958e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0587f4699f2640d1aa4b3035bcb0da9b" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 40031 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "409400b2d53242c699ea0373947ebabd" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 36135 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f12f746012f14fa983ea93b29069958e" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 45985 } health_report { overall_health: HEALTHY } } }
W20260810 04:06:40.921836 1024 tablet.cc:2369] T f782fb179989449e8e992ced2e7ae69a P 0587f4699f2640d1aa4b3035bcb0da9b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 04:06:40.931370 1203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:40.929137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35210 (local address 127.31.32.3:45985)
0810 04:06:40.929189 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0810 04:06:40.929191 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:06:40.929200 (+ 9us) server_negotiation.cc:407] Connection header received
0810 04:06:40.929251 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:06:40.929252 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:06:40.929266 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:06:40.929313 (+ 47us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 04:06:40.929449 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:40.929868 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:40.930602 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:40.930789 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:40.930842 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:06:40.931058 (+ 216us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 04:06:40.931139 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 04:06:40.931194 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0810 04:06:40.931244 (+ 50us) server_negotiation.cc:299] Negotiation successful
0810 04:06:40.931274 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260810 04:06:40.933198 818 raft_consensus.cc:1261] T f782fb179989449e8e992ced2e7ae69a P 409400b2d53242c699ea0373947ebabd [term 1 FOLLOWER]: Refusing update from remote peer f12f746012f14fa983ea93b29069958e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:06:40.933203 973 raft_consensus.cc:1261] T f782fb179989449e8e992ced2e7ae69a P 0587f4699f2640d1aa4b3035bcb0da9b [term 1 FOLLOWER]: Refusing update from remote peer f12f746012f14fa983ea93b29069958e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:06:40.933580 1217 consensus_queue.cc:1035] T f782fb179989449e8e992ced2e7ae69a P f12f746012f14fa983ea93b29069958e [LEADER]: Connected to new peer: Peer: permanent_uuid: "409400b2d53242c699ea0373947ebabd" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 36135 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:06:40.933701 1212 consensus_queue.cc:1035] T f782fb179989449e8e992ced2e7ae69a P f12f746012f14fa983ea93b29069958e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0587f4699f2640d1aa4b3035bcb0da9b" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 40031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:06:40.935949 1220 mvcc.cc:205] Tried to move back new op lower bound from 7316827344619184128 to 7316827344559726592. Current Snapshot: MvccSnapshot[applied={T|T < 7316827344619184128}]
I20260810 04:06:40.940543 639 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33394:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 04:06:40.940691 639 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33394:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 04:06:40.941658 639 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b898b856782d4f13872e6e2373d5a953: Sending DeleteTablet for 3 replicas of tablet f782fb179989449e8e992ced2e7ae69a
I20260810 04:06:40.942164 949 tablet_service.cc:1514] Processing DeleteTablet for tablet f782fb179989449e8e992ced2e7ae69a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:06:40 UTC) from {username='kudu', principal='kudu/127.31.32.62@KRBTEST.COM'} at 127.0.0.1:41090
I20260810 04:06:40.942384 1230 tablet_replica.cc:330] T f782fb179989449e8e992ced2e7ae69a P 0587f4699f2640d1aa4b3035bcb0da9b: stopping tablet replica
I20260810 04:06:40.942384 795 tablet_service.cc:1514] Processing DeleteTablet for tablet f782fb179989449e8e992ced2e7ae69a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:06:40 UTC) from {username='kudu', principal='kudu/127.31.32.62@KRBTEST.COM'} at 127.0.0.1:35874
I20260810 04:06:40.942488 1230 raft_consensus.cc:2229] T f782fb179989449e8e992ced2e7ae69a P 0587f4699f2640d1aa4b3035bcb0da9b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 04:06:40.942638 1109 tablet_service.cc:1514] Processing DeleteTablet for tablet f782fb179989449e8e992ced2e7ae69a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:06:40 UTC) from {username='kudu', principal='kudu/127.31.32.62@KRBTEST.COM'} at 127.0.0.1:35194
I20260810 04:06:40.942880 1232 tablet_replica.cc:330] T f782fb179989449e8e992ced2e7ae69a P f12f746012f14fa983ea93b29069958e: stopping tablet replica
I20260810 04:06:40.942919 1230 raft_consensus.cc:2258] T f782fb179989449e8e992ced2e7ae69a P 0587f4699f2640d1aa4b3035bcb0da9b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:06:40.942880 639 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33394:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 04:06:40.943022 639 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 04:06:40.943181 1232 raft_consensus.cc:2229] T f782fb179989449e8e992ced2e7ae69a P f12f746012f14fa983ea93b29069958e [term 1 LEADER]: Raft consensus shutting down.
I20260810 04:06:40.943325 1232 raft_consensus.cc:2258] T f782fb179989449e8e992ced2e7ae69a P f12f746012f14fa983ea93b29069958e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:06:40.943442 1230 ts_tablet_manager.cc:1905] T f782fb179989449e8e992ced2e7ae69a P 0587f4699f2640d1aa4b3035bcb0da9b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:06:40.943614 1231 tablet_replica.cc:330] T f782fb179989449e8e992ced2e7ae69a P 409400b2d53242c699ea0373947ebabd: stopping tablet replica
I20260810 04:06:40.943703 1231 raft_consensus.cc:2229] T f782fb179989449e8e992ced2e7ae69a P 409400b2d53242c699ea0373947ebabd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 04:06:40.943795 1231 raft_consensus.cc:2258] T f782fb179989449e8e992ced2e7ae69a P 409400b2d53242c699ea0373947ebabd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:06:40.944196 1232 ts_tablet_manager.cc:1905] T f782fb179989449e8e992ced2e7ae69a P f12f746012f14fa983ea93b29069958e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:06:40.944463 949 tablet_service.cc:1467] Processing CreateTablet for tablet 34726e1ac2ac4c3f9374203d18ac729d (DEFAULT_TABLE table=test-table [id=d68264b483de4a8aba2b17ab38f2012a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:06:40.944664 949 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34726e1ac2ac4c3f9374203d18ac729d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:06:40.945488 1230 ts_tablet_manager.cc:1918] T f782fb179989449e8e992ced2e7ae69a P 0587f4699f2640d1aa4b3035bcb0da9b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 04:06:40.945557 1230 log.cc:1196] T f782fb179989449e8e992ced2e7ae69a P 0587f4699f2640d1aa4b3035bcb0da9b: Deleting WAL directory at /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786334794676589-31872-0/minicluster-data/ts-1/wal/wals/f782fb179989449e8e992ced2e7ae69a
I20260810 04:06:40.945729 1232 ts_tablet_manager.cc:1918] T f782fb179989449e8e992ced2e7ae69a P f12f746012f14fa983ea93b29069958e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 04:06:40.945811 1232 log.cc:1196] T f782fb179989449e8e992ced2e7ae69a P f12f746012f14fa983ea93b29069958e: Deleting WAL directory at /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786334794676589-31872-0/minicluster-data/ts-2/wal/wals/f782fb179989449e8e992ced2e7ae69a
I20260810 04:06:40.945813 1230 ts_tablet_manager.cc:1939] T f782fb179989449e8e992ced2e7ae69a P 0587f4699f2640d1aa4b3035bcb0da9b: Deleting consensus metadata
I20260810 04:06:40.945950 1211 tablet_bootstrap.cc:492] T 34726e1ac2ac4c3f9374203d18ac729d P 0587f4699f2640d1aa4b3035bcb0da9b: Bootstrap starting.
I20260810 04:06:40.946003 795 tablet_service.cc:1467] Processing CreateTablet for tablet 34726e1ac2ac4c3f9374203d18ac729d (DEFAULT_TABLE table=test-table [id=d68264b483de4a8aba2b17ab38f2012a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:06:40.946098 1232 ts_tablet_manager.cc:1939] T f782fb179989449e8e992ced2e7ae69a P f12f746012f14fa983ea93b29069958e: Deleting consensus metadata
I20260810 04:06:40.946102 795 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34726e1ac2ac4c3f9374203d18ac729d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:06:40.946393 1211 tablet_bootstrap.cc:654] T 34726e1ac2ac4c3f9374203d18ac729d P 0587f4699f2640d1aa4b3035bcb0da9b: Neither blocks nor log segments found. Creating new log.
I20260810 04:06:40.947108 1211 tablet_bootstrap.cc:492] T 34726e1ac2ac4c3f9374203d18ac729d P 0587f4699f2640d1aa4b3035bcb0da9b: No bootstrap required, opened a new log
I20260810 04:06:40.947155 1211 ts_tablet_manager.cc:1397] T 34726e1ac2ac4c3f9374203d18ac729d P 0587f4699f2640d1aa4b3035bcb0da9b: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260810 04:06:40.947296 1211 raft_consensus.cc:348] T 34726e1ac2ac4c3f9374203d18ac729d P 0587f4699f2640d1aa4b3035bcb0da9b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "409400b2d53242c699ea0373947ebabd" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 36135 } } peers { permanent_uuid: "0587f4699f2640d1aa4b3035bcb0da9b" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 40031 } } peers { permanent_uuid: "f12f746012f14fa983ea93b29069958e" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 45985 } }
I20260810 04:06:40.947366 1211 raft_consensus.cc:374] T 34726e1ac2ac4c3f9374203d18ac729d P 0587f4699f2640d1aa4b3035bcb0da9b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:06:40.947388 1211 raft_consensus.cc:729] T 34726e1ac2ac4c3f9374203d18ac729d P 0587f4699f2640d1aa4b3035bcb0da9b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0587f4699f2640d1aa4b3035bcb0da9b, State: Initialized, Role: FOLLOWER
I20260810 04:06:40.947431 1211 consensus_queue.cc:260] T 34726e1ac2ac4c3f9374203d18ac729d P 0587f4699f2640d1aa4b3035bcb0da9b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "409400b2d53242c699ea0373947ebabd" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 36135 } } peers { permanent_uuid: "0587f4699f2640d1aa4b3035bcb0da9b" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 40031 } } peers { permanent_uuid: "f12f746012f14fa983ea93b29069958e" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 45985 } }
I20260810 04:06:40.947511 1211 ts_tablet_manager.cc:1428] T 34726e1ac2ac4c3f9374203d18ac729d P 0587f4699f2640d1aa4b3035bcb0da9b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260810 04:06:40.947531 1206 tablet_bootstrap.cc:492] T 34726e1ac2ac4c3f9374203d18ac729d P 409400b2d53242c699ea0373947ebabd: Bootstrap starting.
I20260810 04:06:40.947574 1109 tablet_service.cc:1467] Processing CreateTablet for tablet 34726e1ac2ac4c3f9374203d18ac729d (DEFAULT_TABLE table=test-table [id=d68264b483de4a8aba2b17ab38f2012a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:06:40.947667 1109 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34726e1ac2ac4c3f9374203d18ac729d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:06:40.947937 1206 tablet_bootstrap.cc:654] T 34726e1ac2ac4c3f9374203d18ac729d P 409400b2d53242c699ea0373947ebabd: Neither blocks nor log segments found. Creating new log.
I20260810 04:06:40.948596 1206 tablet_bootstrap.cc:492] T 34726e1ac2ac4c3f9374203d18ac729d P 409400b2d53242c699ea0373947ebabd: No bootstrap required, opened a new log
I20260810 04:06:40.948647 1206 ts_tablet_manager.cc:1397] T 34726e1ac2ac4c3f9374203d18ac729d P 409400b2d53242c699ea0373947ebabd: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260810 04:06:40.948778 1206 raft_consensus.cc:348] T 34726e1ac2ac4c3f9374203d18ac729d P 409400b2d53242c699ea0373947ebabd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "409400b2d53242c699ea0373947ebabd" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 36135 } } peers { permanent_uuid: "0587f4699f2640d1aa4b3035bcb0da9b" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 40031 } } peers { permanent_uuid: "f12f746012f14fa983ea93b29069958e" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 45985 } }
I20260810 04:06:40.948838 1206 raft_consensus.cc:374] T 34726e1ac2ac4c3f9374203d18ac729d P 409400b2d53242c699ea0373947ebabd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:06:40.948863 1206 raft_consensus.cc:729] T 34726e1ac2ac4c3f9374203d18ac729d P 409400b2d53242c699ea0373947ebabd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 409400b2d53242c699ea0373947ebabd, State: Initialized, Role: FOLLOWER
I20260810 04:06:40.948891 1206 consensus_queue.cc:260] T 34726e1ac2ac4c3f9374203d18ac729d P 409400b2d53242c699ea0373947ebabd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "409400b2d53242c699ea0373947ebabd" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 36135 } } peers { permanent_uuid: "0587f4699f2640d1aa4b3035bcb0da9b" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 40031 } } peers { permanent_uuid: "f12f746012f14fa983ea93b29069958e" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 45985 } }
I20260810 04:06:40.948925 1207 tablet_bootstrap.cc:492] T 34726e1ac2ac4c3f9374203d18ac729d P f12f746012f14fa983ea93b29069958e: Bootstrap starting.
I20260810 04:06:40.948962 1206 ts_tablet_manager.cc:1428] T 34726e1ac2ac4c3f9374203d18ac729d P 409400b2d53242c699ea0373947ebabd: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260810 04:06:40.949357 1207 tablet_bootstrap.cc:654] T 34726e1ac2ac4c3f9374203d18ac729d P f12f746012f14fa983ea93b29069958e: Neither blocks nor log segments found. Creating new log.
I20260810 04:06:40.949453 620 catalog_manager.cc:4841] TS 0587f4699f2640d1aa4b3035bcb0da9b (127.31.32.2:40031): tablet f782fb179989449e8e992ced2e7ae69a (table test-table [id=d02ba4a0cb644823a8f76fd57eab7361]) successfully deleted
I20260810 04:06:40.949529 624 catalog_manager.cc:4841] TS f12f746012f14fa983ea93b29069958e (127.31.32.3:45985): tablet f782fb179989449e8e992ced2e7ae69a (table test-table [id=d02ba4a0cb644823a8f76fd57eab7361]) successfully deleted
I20260810 04:06:40.950141 1207 tablet_bootstrap.cc:492] T 34726e1ac2ac4c3f9374203d18ac729d P f12f746012f14fa983ea93b29069958e: No bootstrap required, opened a new log
I20260810 04:06:40.950194 1207 ts_tablet_manager.cc:1397] T 34726e1ac2ac4c3f9374203d18ac729d P f12f746012f14fa983ea93b29069958e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260810 04:06:40.950203 1231 ts_tablet_manager.cc:1905] T f782fb179989449e8e992ced2e7ae69a P 409400b2d53242c699ea0373947ebabd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:06:40.950350 1207 raft_consensus.cc:348] T 34726e1ac2ac4c3f9374203d18ac729d P f12f746012f14fa983ea93b29069958e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "409400b2d53242c699ea0373947ebabd" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 36135 } } peers { permanent_uuid: "0587f4699f2640d1aa4b3035bcb0da9b" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 40031 } } peers { permanent_uuid: "f12f746012f14fa983ea93b29069958e" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 45985 } }
I20260810 04:06:40.950426 1207 raft_consensus.cc:374] T 34726e1ac2ac4c3f9374203d18ac729d P f12f746012f14fa983ea93b29069958e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:06:40.950446 1207 raft_consensus.cc:729] T 34726e1ac2ac4c3f9374203d18ac729d P f12f746012f14fa983ea93b29069958e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f12f746012f14fa983ea93b29069958e, State: Initialized, Role: FOLLOWER
I20260810 04:06:40.950493 1207 consensus_queue.cc:260] T 34726e1ac2ac4c3f9374203d18ac729d P f12f746012f14fa983ea93b29069958e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "409400b2d53242c699ea0373947ebabd" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 36135 } } peers { permanent_uuid: "0587f4699f2640d1aa4b3035bcb0da9b" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 40031 } } peers { permanent_uuid: "f12f746012f14fa983ea93b29069958e" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 45985 } }
I20260810 04:06:40.950587 1207 ts_tablet_manager.cc:1428] T 34726e1ac2ac4c3f9374203d18ac729d P f12f746012f14fa983ea93b29069958e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260810 04:06:40.951546 1231 ts_tablet_manager.cc:1918] T f782fb179989449e8e992ced2e7ae69a P 409400b2d53242c699ea0373947ebabd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 04:06:40.951602 1231 log.cc:1196] T f782fb179989449e8e992ced2e7ae69a P 409400b2d53242c699ea0373947ebabd: Deleting WAL directory at /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786334794676589-31872-0/minicluster-data/ts-0/wal/wals/f782fb179989449e8e992ced2e7ae69a
I20260810 04:06:40.951772 1231 ts_tablet_manager.cc:1939] T f782fb179989449e8e992ced2e7ae69a P 409400b2d53242c699ea0373947ebabd: Deleting consensus metadata
I20260810 04:06:40.952189 624 catalog_manager.cc:4841] TS 409400b2d53242c699ea0373947ebabd (127.31.32.1:36135): tablet f782fb179989449e8e992ced2e7ae69a (table test-table [id=d02ba4a0cb644823a8f76fd57eab7361]) successfully deleted
I20260810 04:06:40.975068 1215 raft_consensus.cc:482] T 34726e1ac2ac4c3f9374203d18ac729d P 0587f4699f2640d1aa4b3035bcb0da9b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 04:06:40.975157 1215 raft_consensus.cc:504] T 34726e1ac2ac4c3f9374203d18ac729d P 0587f4699f2640d1aa4b3035bcb0da9b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "409400b2d53242c699ea0373947ebabd" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 36135 } } peers { permanent_uuid: "0587f4699f2640d1aa4b3035bcb0da9b" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 40031 } } peers { permanent_uuid: "f12f746012f14fa983ea93b29069958e" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 45985 } }
I20260810 04:06:40.975543 1215 leader_election.cc:288] T 34726e1ac2ac4c3f9374203d18ac729d P 0587f4699f2640d1aa4b3035bcb0da9b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 409400b2d53242c699ea0373947ebabd (127.31.32.1:36135), f12f746012f14fa983ea93b29069958e (127.31.32.3:45985)
I20260810 04:06:40.978989 906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:40.975727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.1:36135 (local address 127.31.32.2:54227)
0810 04:06:40.975954 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0810 04:06:40.975966 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 04:06:40.976034 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:06:40.976226 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:06:40.976227 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:06:40.976228 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:06:40.976307 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:06:40.976309 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:06:40.977291 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:06:40.977292 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:06:40.978075 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:06:40.978078 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:06:40.978692 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:06:40.978692 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:06:40.978798 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:06:40.978802 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 04:06:40.978859 (+ 57us) client_negotiation.cc:240] Negotiation successful
0810 04:06:40.978899 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207}
I20260810 04:06:40.979000 1201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:40.975792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.2:54227 (local address 127.31.32.1:36135)
0810 04:06:40.975839 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0810 04:06:40.975841 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:06:40.976092 (+ 251us) server_negotiation.cc:407] Connection header received
0810 04:06:40.976142 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:06:40.976143 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:06:40.976153 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:06:40.976203 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:06:40.976385 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:40.977203 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:40.978194 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:40.978555 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:40.978647 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:06:40.978659 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0810 04:06:40.978885 (+ 226us) server_negotiation.cc:299] Negotiation successful
0810 04:06:40.978933 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260810 04:06:40.979354 818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34726e1ac2ac4c3f9374203d18ac729d" candidate_uuid: "0587f4699f2640d1aa4b3035bcb0da9b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "409400b2d53242c699ea0373947ebabd" is_pre_election: true
I20260810 04:06:40.979444 818 raft_consensus.cc:2454] T 34726e1ac2ac4c3f9374203d18ac729d P 409400b2d53242c699ea0373947ebabd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0587f4699f2640d1aa4b3035bcb0da9b in term 0.
I20260810 04:06:40.979594 1237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:40.975836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.3:45985 (local address 127.31.32.2:52435)
0810 04:06:40.975974 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0810 04:06:40.975985 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 04:06:40.976053 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:06:40.976265 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:06:40.976266 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:06:40.976267 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:06:40.976366 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:06:40.976367 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:06:40.977244 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:06:40.977245 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:06:40.978520 (+ 1275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:06:40.978523 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:06:40.979325 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:06:40.979327 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:06:40.979401 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:06:40.979404 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 04:06:40.979480 (+ 76us) client_negotiation.cc:240] Negotiation successful
0810 04:06:40.979519 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":81,"threads_started":1}
I20260810 04:06:40.979663 1203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:40.976084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.2:52435 (local address 127.31.32.3:45985)
0810 04:06:40.976147 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0810 04:06:40.976149 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:06:40.976157 (+ 8us) server_negotiation.cc:407] Connection header received
0810 04:06:40.976181 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:06:40.976182 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:06:40.976190 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:06:40.976239 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:06:40.976431 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:40.977155 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:40.978620 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:40.979185 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:40.979242 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:06:40.979252 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0810 04:06:40.979552 (+ 300us) server_negotiation.cc:299] Negotiation successful
0810 04:06:40.979591 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260810 04:06:40.979604 900 leader_election.cc:302] T 34726e1ac2ac4c3f9374203d18ac729d P 0587f4699f2640d1aa4b3035bcb0da9b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0587f4699f2640d1aa4b3035bcb0da9b, 409400b2d53242c699ea0373947ebabd; no voters:
I20260810 04:06:40.979820 1215 raft_consensus.cc:2781] T 34726e1ac2ac4c3f9374203d18ac729d P 0587f4699f2640d1aa4b3035bcb0da9b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 04:06:40.979863 1215 raft_consensus.cc:482] T 34726e1ac2ac4c3f9374203d18ac729d P 0587f4699f2640d1aa4b3035bcb0da9b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 04:06:40.979889 1215 raft_consensus.cc:3037] T 34726e1ac2ac4c3f9374203d18ac729d P 0587f4699f2640d1aa4b3035bcb0da9b [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:06:40.979933 1131 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34726e1ac2ac4c3f9374203d18ac729d" candidate_uuid: "0587f4699f2640d1aa4b3035bcb0da9b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f12f746012f14fa983ea93b29069958e" is_pre_election: true
I20260810 04:06:40.980058 1131 raft_consensus.cc:2454] T 34726e1ac2ac4c3f9374203d18ac729d P f12f746012f14fa983ea93b29069958e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0587f4699f2640d1aa4b3035bcb0da9b in term 0.
I20260810 04:06:40.980715 1215 raft_consensus.cc:504] T 34726e1ac2ac4c3f9374203d18ac729d P 0587f4699f2640d1aa4b3035bcb0da9b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "409400b2d53242c699ea0373947ebabd" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 36135 } } peers { permanent_uuid: "0587f4699f2640d1aa4b3035bcb0da9b" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 40031 } } peers { permanent_uuid: "f12f746012f14fa983ea93b29069958e" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 45985 } }
I20260810 04:06:40.980855 1215 leader_election.cc:288] T 34726e1ac2ac4c3f9374203d18ac729d P 0587f4699f2640d1aa4b3035bcb0da9b [CANDIDATE]: Term 1 election: Requested vote from peers 409400b2d53242c699ea0373947ebabd (127.31.32.1:36135), f12f746012f14fa983ea93b29069958e (127.31.32.3:45985)
I20260810 04:06:40.980988 818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34726e1ac2ac4c3f9374203d18ac729d" candidate_uuid: "0587f4699f2640d1aa4b3035bcb0da9b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "409400b2d53242c699ea0373947ebabd"
I20260810 04:06:40.981070 818 raft_consensus.cc:3037] T 34726e1ac2ac4c3f9374203d18ac729d P 409400b2d53242c699ea0373947ebabd [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:06:40.981787 818 raft_consensus.cc:2454] T 34726e1ac2ac4c3f9374203d18ac729d P 409400b2d53242c699ea0373947ebabd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0587f4699f2640d1aa4b3035bcb0da9b in term 1.
I20260810 04:06:40.981895 1131 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34726e1ac2ac4c3f9374203d18ac729d" candidate_uuid: "0587f4699f2640d1aa4b3035bcb0da9b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f12f746012f14fa983ea93b29069958e"
I20260810 04:06:40.981963 1131 raft_consensus.cc:3037] T 34726e1ac2ac4c3f9374203d18ac729d P f12f746012f14fa983ea93b29069958e [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:06:40.981992 900 leader_election.cc:302] T 34726e1ac2ac4c3f9374203d18ac729d P 0587f4699f2640d1aa4b3035bcb0da9b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0587f4699f2640d1aa4b3035bcb0da9b, 409400b2d53242c699ea0373947ebabd; no voters:
I20260810 04:06:40.982079 1215 raft_consensus.cc:2781] T 34726e1ac2ac4c3f9374203d18ac729d P 0587f4699f2640d1aa4b3035bcb0da9b [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:06:40.982227 1215 raft_consensus.cc:686] T 34726e1ac2ac4c3f9374203d18ac729d P 0587f4699f2640d1aa4b3035bcb0da9b [term 1 LEADER]: Becoming Leader. State: Replica: 0587f4699f2640d1aa4b3035bcb0da9b, State: Running, Role: LEADER
I20260810 04:06:40.982303 1215 consensus_queue.cc:237] T 34726e1ac2ac4c3f9374203d18ac729d P 0587f4699f2640d1aa4b3035bcb0da9b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "409400b2d53242c699ea0373947ebabd" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 36135 } } peers { permanent_uuid: "0587f4699f2640d1aa4b3035bcb0da9b" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 40031 } } peers { permanent_uuid: "f12f746012f14fa983ea93b29069958e" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 45985 } }
I20260810 04:06:40.982730 1131 raft_consensus.cc:2454] T 34726e1ac2ac4c3f9374203d18ac729d P f12f746012f14fa983ea93b29069958e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0587f4699f2640d1aa4b3035bcb0da9b in term 1.
I20260810 04:06:40.983232 641 catalog_manager.cc:5495] T 34726e1ac2ac4c3f9374203d18ac729d P 0587f4699f2640d1aa4b3035bcb0da9b reported cstate change: term changed from 0 to 1, leader changed from <none> to 0587f4699f2640d1aa4b3035bcb0da9b (127.31.32.2). New cstate: current_term: 1 leader_uuid: "0587f4699f2640d1aa4b3035bcb0da9b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "409400b2d53242c699ea0373947ebabd" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 36135 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0587f4699f2640d1aa4b3035bcb0da9b" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 40031 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f12f746012f14fa983ea93b29069958e" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 45985 } health_report { overall_health: UNKNOWN } } }
W20260810 04:06:41.021014 871 tablet.cc:2369] T 34726e1ac2ac4c3f9374203d18ac729d P 409400b2d53242c699ea0373947ebabd: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 10 04:06:41 dist-test-slave-8w9v krb5kdc[589](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334800, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.62@KRBTEST.COM for kudu/127.31.32.62@KRBTEST.COM
I20260810 04:06:41.025511 1248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:41.016930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:38035 (local address 127.0.0.1:33408)
0810 04:06:41.017125 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0810 04:06:41.017138 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 04:06:41.017224 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:06:41.017389 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:06:41.017390 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:06:41.017807 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:06:41.017869 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:06:41.017872 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:06:41.018533 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:06:41.018535 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:06:41.019386 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:06:41.019389 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:06:41.019733 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:06:41.019734 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:06:41.019797 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:06:41.020508 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:06:41.020514 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:06:41.022308 (+ 1794us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:06:41.024430 (+ 2122us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:06:41.024436 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:06:41.024438 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:06:41.024673 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:06:41.024895 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:06:41.024896 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:06:41.024896 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:06:41.024950 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:06:41.025221 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:06:41.025227 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:06:41.025284 (+ 57us) client_negotiation.cc:769] Sending connection context
0810 04:06:41.025342 (+ 58us) client_negotiation.cc:240] Negotiation successful
0810 04:06:41.025438 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":103,"threads_started":1}
I20260810 04:06:41.025619 696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:41.017042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33408 (local address 127.31.32.62:38035)
0810 04:06:41.017095 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0810 04:06:41.017097 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:06:41.017217 (+ 120us) server_negotiation.cc:407] Connection header received
0810 04:06:41.017287 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:06:41.017288 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:06:41.017297 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:06:41.017362 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:06:41.017949 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:41.018445 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:41.019472 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:41.019649 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:41.019700 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:06:41.022427 (+ 2727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:06:41.022428 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:06:41.022429 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:06:41.022435 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:06:41.024331 (+ 1896us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:06:41.024756 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:06:41.024757 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:06:41.024758 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:06:41.024822 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:06:41.025015 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:06:41.025016 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:06:41.025016 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:06:41.025154 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:06:41.025227 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0810 04:06:41.025399 (+ 172us) server_negotiation.cc:299] Negotiation successful
0810 04:06:41.025508 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":29}
I20260810 04:06:41.027799 641 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.31.32.62@KRBTEST.COM'} at 127.0.0.1:33408:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260810 04:06:41.028096 641 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 04:06:41.029517 795 tablet_service.cc:1467] Processing CreateTablet for tablet f8c964ce460a4cbba491d8cd72cbc65e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9a5e8d8fe9654f82a5739b0225aa191e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 04:06:41.029596 1109 tablet_service.cc:1467] Processing CreateTablet for tablet f8c964ce460a4cbba491d8cd72cbc65e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9a5e8d8fe9654f82a5739b0225aa191e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 04:06:41.029748 1109 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8c964ce460a4cbba491d8cd72cbc65e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:06:41.029750 795 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8c964ce460a4cbba491d8cd72cbc65e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:06:41.031103 1206 tablet_bootstrap.cc:492] T f8c964ce460a4cbba491d8cd72cbc65e P 409400b2d53242c699ea0373947ebabd: Bootstrap starting.
I20260810 04:06:41.031172 949 tablet_service.cc:1467] Processing CreateTablet for tablet f8c964ce460a4cbba491d8cd72cbc65e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9a5e8d8fe9654f82a5739b0225aa191e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 04:06:41.031306 949 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8c964ce460a4cbba491d8cd72cbc65e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:06:41.031369 1207 tablet_bootstrap.cc:492] T f8c964ce460a4cbba491d8cd72cbc65e P f12f746012f14fa983ea93b29069958e: Bootstrap starting.
I20260810 04:06:41.031503 1206 tablet_bootstrap.cc:654] T f8c964ce460a4cbba491d8cd72cbc65e P 409400b2d53242c699ea0373947ebabd: Neither blocks nor log segments found. Creating new log.
I20260810 04:06:41.031952 1207 tablet_bootstrap.cc:654] T f8c964ce460a4cbba491d8cd72cbc65e P f12f746012f14fa983ea93b29069958e: Neither blocks nor log segments found. Creating new log.
I20260810 04:06:41.032306 1206 tablet_bootstrap.cc:492] T f8c964ce460a4cbba491d8cd72cbc65e P 409400b2d53242c699ea0373947ebabd: No bootstrap required, opened a new log
I20260810 04:06:41.032353 1206 ts_tablet_manager.cc:1397] T f8c964ce460a4cbba491d8cd72cbc65e P 409400b2d53242c699ea0373947ebabd: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260810 04:06:41.032505 1206 raft_consensus.cc:348] T f8c964ce460a4cbba491d8cd72cbc65e P 409400b2d53242c699ea0373947ebabd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "409400b2d53242c699ea0373947ebabd" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 36135 } } peers { permanent_uuid: "0587f4699f2640d1aa4b3035bcb0da9b" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 40031 } } peers { permanent_uuid: "f12f746012f14fa983ea93b29069958e" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 45985 } }
I20260810 04:06:41.032608 1206 raft_consensus.cc:374] T f8c964ce460a4cbba491d8cd72cbc65e P 409400b2d53242c699ea0373947ebabd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:06:41.032631 1206 raft_consensus.cc:729] T f8c964ce460a4cbba491d8cd72cbc65e P 409400b2d53242c699ea0373947ebabd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 409400b2d53242c699ea0373947ebabd, State: Initialized, Role: FOLLOWER
I20260810 04:06:41.032672 1206 consensus_queue.cc:260] T f8c964ce460a4cbba491d8cd72cbc65e P 409400b2d53242c699ea0373947ebabd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "409400b2d53242c699ea0373947ebabd" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 36135 } } peers { permanent_uuid: "0587f4699f2640d1aa4b3035bcb0da9b" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 40031 } } peers { permanent_uuid: "f12f746012f14fa983ea93b29069958e" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 45985 } }
I20260810 04:06:41.032753 1206 ts_tablet_manager.cc:1428] T f8c964ce460a4cbba491d8cd72cbc65e P 409400b2d53242c699ea0373947ebabd: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260810 04:06:41.032922 1207 tablet_bootstrap.cc:492] T f8c964ce460a4cbba491d8cd72cbc65e P f12f746012f14fa983ea93b29069958e: No bootstrap required, opened a new log
I20260810 04:06:41.032968 1207 ts_tablet_manager.cc:1397] T f8c964ce460a4cbba491d8cd72cbc65e P f12f746012f14fa983ea93b29069958e: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260810 04:06:41.032900 1210 tablet_replica.cc:439] T f8c964ce460a4cbba491d8cd72cbc65e P 409400b2d53242c699ea0373947ebabd: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "409400b2d53242c699ea0373947ebabd" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 36135 } } peers { permanent_uuid: "0587f4699f2640d1aa4b3035bcb0da9b" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 40031 } } peers { permanent_uuid: "f12f746012f14fa983ea93b29069958e" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 45985 } } }
I20260810 04:06:41.033003 1210 tablet_replica.cc:442] T f8c964ce460a4cbba491d8cd72cbc65e P 409400b2d53242c699ea0373947ebabd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 04:06:41.033100 1207 raft_consensus.cc:348] T f8c964ce460a4cbba491d8cd72cbc65e P f12f746012f14fa983ea93b29069958e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "409400b2d53242c699ea0373947ebabd" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 36135 } } peers { permanent_uuid: "0587f4699f2640d1aa4b3035bcb0da9b" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 40031 } } peers { permanent_uuid: "f12f746012f14fa983ea93b29069958e" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 45985 } }
I20260810 04:06:41.033165 1207 raft_consensus.cc:374] T f8c964ce460a4cbba491d8cd72cbc65e P f12f746012f14fa983ea93b29069958e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:06:41.033188 1207 raft_consensus.cc:729] T f8c964ce460a4cbba491d8cd72cbc65e P f12f746012f14fa983ea93b29069958e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f12f746012f14fa983ea93b29069958e, State: Initialized, Role: FOLLOWER
I20260810 04:06:41.033231 1207 consensus_queue.cc:260] T f8c964ce460a4cbba491d8cd72cbc65e P f12f746012f14fa983ea93b29069958e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "409400b2d53242c699ea0373947ebabd" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 36135 } } peers { permanent_uuid: "0587f4699f2640d1aa4b3035bcb0da9b" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 40031 } } peers { permanent_uuid: "f12f746012f14fa983ea93b29069958e" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 45985 } }
I20260810 04:06:41.033313 1207 ts_tablet_manager.cc:1428] T f8c964ce460a4cbba491d8cd72cbc65e P f12f746012f14fa983ea93b29069958e: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260810 04:06:41.033349 1211 tablet_bootstrap.cc:492] T f8c964ce460a4cbba491d8cd72cbc65e P 0587f4699f2640d1aa4b3035bcb0da9b: Bootstrap starting.
I20260810 04:06:41.033437 1226 tablet_replica.cc:439] T f8c964ce460a4cbba491d8cd72cbc65e P f12f746012f14fa983ea93b29069958e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "409400b2d53242c699ea0373947ebabd" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 36135 } } peers { permanent_uuid: "0587f4699f2640d1aa4b3035bcb0da9b" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 40031 } } peers { permanent_uuid: "f12f746012f14fa983ea93b29069958e" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 45985 } } }
I20260810 04:06:41.033504 1226 tablet_replica.cc:442] T f8c964ce460a4cbba491d8cd72cbc65e P f12f746012f14fa983ea93b29069958e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 04:06:41.033753 1211 tablet_bootstrap.cc:654] T f8c964ce460a4cbba491d8cd72cbc65e P 0587f4699f2640d1aa4b3035bcb0da9b: Neither blocks nor log segments found. Creating new log.
I20260810 04:06:41.034425 1211 tablet_bootstrap.cc:492] T f8c964ce460a4cbba491d8cd72cbc65e P 0587f4699f2640d1aa4b3035bcb0da9b: No bootstrap required, opened a new log
I20260810 04:06:41.034478 1211 ts_tablet_manager.cc:1397] T f8c964ce460a4cbba491d8cd72cbc65e P 0587f4699f2640d1aa4b3035bcb0da9b: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260810 04:06:41.034626 1211 raft_consensus.cc:348] T f8c964ce460a4cbba491d8cd72cbc65e P 0587f4699f2640d1aa4b3035bcb0da9b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "409400b2d53242c699ea0373947ebabd" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 36135 } } peers { permanent_uuid: "0587f4699f2640d1aa4b3035bcb0da9b" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 40031 } } peers { permanent_uuid: "f12f746012f14fa983ea93b29069958e" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 45985 } }
I20260810 04:06:41.034698 1211 raft_consensus.cc:374] T f8c964ce460a4cbba491d8cd72cbc65e P 0587f4699f2640d1aa4b3035bcb0da9b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:06:41.034725 1211 raft_consensus.cc:729] T f8c964ce460a4cbba491d8cd72cbc65e P 0587f4699f2640d1aa4b3035bcb0da9b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0587f4699f2640d1aa4b3035bcb0da9b, State: Initialized, Role: FOLLOWER
I20260810 04:06:41.034768 1211 consensus_queue.cc:260] T f8c964ce460a4cbba491d8cd72cbc65e P 0587f4699f2640d1aa4b3035bcb0da9b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "409400b2d53242c699ea0373947ebabd" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 36135 } } peers { permanent_uuid: "0587f4699f2640d1aa4b3035bcb0da9b" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 40031 } } peers { permanent_uuid: "f12f746012f14fa983ea93b29069958e" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 45985 } }
I20260810 04:06:41.034864 1211 ts_tablet_manager.cc:1428] T f8c964ce460a4cbba491d8cd72cbc65e P 0587f4699f2640d1aa4b3035bcb0da9b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260810 04:06:41.034905 1215 tablet_replica.cc:439] T f8c964ce460a4cbba491d8cd72cbc65e P 0587f4699f2640d1aa4b3035bcb0da9b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "409400b2d53242c699ea0373947ebabd" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 36135 } } peers { permanent_uuid: "0587f4699f2640d1aa4b3035bcb0da9b" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 40031 } } peers { permanent_uuid: "f12f746012f14fa983ea93b29069958e" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 45985 } } }
I20260810 04:06:41.034970 1215 tablet_replica.cc:442] T f8c964ce460a4cbba491d8cd72cbc65e P 0587f4699f2640d1aa4b3035bcb0da9b: This TxnStatusTablet replica's current role is: FOLLOWER
W20260810 04:06:41.094394 1181 tablet.cc:2369] T 34726e1ac2ac4c3f9374203d18ac729d P f12f746012f14fa983ea93b29069958e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 04:06:41.174504 1215 raft_consensus.cc:482] T f8c964ce460a4cbba491d8cd72cbc65e P 0587f4699f2640d1aa4b3035bcb0da9b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 04:06:41.174609 1215 raft_consensus.cc:504] T f8c964ce460a4cbba491d8cd72cbc65e P 0587f4699f2640d1aa4b3035bcb0da9b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "409400b2d53242c699ea0373947ebabd" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 36135 } } peers { permanent_uuid: "0587f4699f2640d1aa4b3035bcb0da9b" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 40031 } } peers { permanent_uuid: "f12f746012f14fa983ea93b29069958e" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 45985 } }
I20260810 04:06:41.174768 1215 leader_election.cc:288] T f8c964ce460a4cbba491d8cd72cbc65e P 0587f4699f2640d1aa4b3035bcb0da9b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 409400b2d53242c699ea0373947ebabd (127.31.32.1:36135), f12f746012f14fa983ea93b29069958e (127.31.32.3:45985)
I20260810 04:06:41.174988 818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8c964ce460a4cbba491d8cd72cbc65e" candidate_uuid: "0587f4699f2640d1aa4b3035bcb0da9b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "409400b2d53242c699ea0373947ebabd" is_pre_election: true
I20260810 04:06:41.175067 1131 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8c964ce460a4cbba491d8cd72cbc65e" candidate_uuid: "0587f4699f2640d1aa4b3035bcb0da9b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f12f746012f14fa983ea93b29069958e" is_pre_election: true
I20260810 04:06:41.175129 818 raft_consensus.cc:2454] T f8c964ce460a4cbba491d8cd72cbc65e P 409400b2d53242c699ea0373947ebabd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0587f4699f2640d1aa4b3035bcb0da9b in term 0.
I20260810 04:06:41.175151 1131 raft_consensus.cc:2454] T f8c964ce460a4cbba491d8cd72cbc65e P f12f746012f14fa983ea93b29069958e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0587f4699f2640d1aa4b3035bcb0da9b in term 0.
I20260810 04:06:41.175307 900 leader_election.cc:302] T f8c964ce460a4cbba491d8cd72cbc65e P 0587f4699f2640d1aa4b3035bcb0da9b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0587f4699f2640d1aa4b3035bcb0da9b, f12f746012f14fa983ea93b29069958e; no voters:
I20260810 04:06:41.175417 1215 raft_consensus.cc:2781] T f8c964ce460a4cbba491d8cd72cbc65e P 0587f4699f2640d1aa4b3035bcb0da9b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 04:06:41.175458 1215 raft_consensus.cc:482] T f8c964ce460a4cbba491d8cd72cbc65e P 0587f4699f2640d1aa4b3035bcb0da9b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 04:06:41.175477 1215 raft_consensus.cc:3037] T f8c964ce460a4cbba491d8cd72cbc65e P 0587f4699f2640d1aa4b3035bcb0da9b [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:06:41.176196 1215 raft_consensus.cc:504] T f8c964ce460a4cbba491d8cd72cbc65e P 0587f4699f2640d1aa4b3035bcb0da9b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "409400b2d53242c699ea0373947ebabd" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 36135 } } peers { permanent_uuid: "0587f4699f2640d1aa4b3035bcb0da9b" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 40031 } } peers { permanent_uuid: "f12f746012f14fa983ea93b29069958e" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 45985 } }
I20260810 04:06:41.176332 1215 leader_election.cc:288] T f8c964ce460a4cbba491d8cd72cbc65e P 0587f4699f2640d1aa4b3035bcb0da9b [CANDIDATE]: Term 1 election: Requested vote from peers 409400b2d53242c699ea0373947ebabd (127.31.32.1:36135), f12f746012f14fa983ea93b29069958e (127.31.32.3:45985)
I20260810 04:06:41.176537 1131 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8c964ce460a4cbba491d8cd72cbc65e" candidate_uuid: "0587f4699f2640d1aa4b3035bcb0da9b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f12f746012f14fa983ea93b29069958e"
I20260810 04:06:41.176517 818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8c964ce460a4cbba491d8cd72cbc65e" candidate_uuid: "0587f4699f2640d1aa4b3035bcb0da9b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "409400b2d53242c699ea0373947ebabd"
I20260810 04:06:41.176626 818 raft_consensus.cc:3037] T f8c964ce460a4cbba491d8cd72cbc65e P 409400b2d53242c699ea0373947ebabd [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:06:41.176638 1131 raft_consensus.cc:3037] T f8c964ce460a4cbba491d8cd72cbc65e P f12f746012f14fa983ea93b29069958e [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:06:41.177354 818 raft_consensus.cc:2454] T f8c964ce460a4cbba491d8cd72cbc65e P 409400b2d53242c699ea0373947ebabd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0587f4699f2640d1aa4b3035bcb0da9b in term 1.
I20260810 04:06:41.177402 1131 raft_consensus.cc:2454] T f8c964ce460a4cbba491d8cd72cbc65e P f12f746012f14fa983ea93b29069958e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0587f4699f2640d1aa4b3035bcb0da9b in term 1.
I20260810 04:06:41.177557 900 leader_election.cc:302] T f8c964ce460a4cbba491d8cd72cbc65e P 0587f4699f2640d1aa4b3035bcb0da9b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0587f4699f2640d1aa4b3035bcb0da9b, f12f746012f14fa983ea93b29069958e; no voters:
I20260810 04:06:41.177668 1215 raft_consensus.cc:2781] T f8c964ce460a4cbba491d8cd72cbc65e P 0587f4699f2640d1aa4b3035bcb0da9b [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:06:41.177728 1215 raft_consensus.cc:686] T f8c964ce460a4cbba491d8cd72cbc65e P 0587f4699f2640d1aa4b3035bcb0da9b [term 1 LEADER]: Becoming Leader. State: Replica: 0587f4699f2640d1aa4b3035bcb0da9b, State: Running, Role: LEADER
I20260810 04:06:41.177798 1215 consensus_queue.cc:237] T f8c964ce460a4cbba491d8cd72cbc65e P 0587f4699f2640d1aa4b3035bcb0da9b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "409400b2d53242c699ea0373947ebabd" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 36135 } } peers { permanent_uuid: "0587f4699f2640d1aa4b3035bcb0da9b" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 40031 } } peers { permanent_uuid: "f12f746012f14fa983ea93b29069958e" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 45985 } }
I20260810 04:06:41.178103 1238 tablet_replica.cc:439] T f8c964ce460a4cbba491d8cd72cbc65e P 0587f4699f2640d1aa4b3035bcb0da9b: TxnStatusTablet state changed. Reason: New leader 0587f4699f2640d1aa4b3035bcb0da9b. Latest consensus state: current_term: 1 leader_uuid: "0587f4699f2640d1aa4b3035bcb0da9b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "409400b2d53242c699ea0373947ebabd" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 36135 } } peers { permanent_uuid: "0587f4699f2640d1aa4b3035bcb0da9b" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 40031 } } peers { permanent_uuid: "f12f746012f14fa983ea93b29069958e" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 45985 } } }
I20260810 04:06:41.178205 1238 tablet_replica.cc:442] T f8c964ce460a4cbba491d8cd72cbc65e P 0587f4699f2640d1aa4b3035bcb0da9b: This TxnStatusTablet replica's current role is: LEADER
I20260810 04:06:41.178400 641 catalog_manager.cc:5495] T f8c964ce460a4cbba491d8cd72cbc65e P 0587f4699f2640d1aa4b3035bcb0da9b reported cstate change: term changed from 0 to 1, leader changed from <none> to 0587f4699f2640d1aa4b3035bcb0da9b (127.31.32.2). New cstate: current_term: 1 leader_uuid: "0587f4699f2640d1aa4b3035bcb0da9b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "409400b2d53242c699ea0373947ebabd" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 36135 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0587f4699f2640d1aa4b3035bcb0da9b" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 40031 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f12f746012f14fa983ea93b29069958e" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 45985 } health_report { overall_health: UNKNOWN } } }
I20260810 04:06:41.178763 1254 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260810 04:06:41.178915 1254 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260810 04:06:41.218011 1248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:41.215322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.2:40031 (local address 127.0.0.1:41100)
0810 04:06:41.215490 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0810 04:06:41.215501 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 04:06:41.215560 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:06:41.215746 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:06:41.215746 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:06:41.215748 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 04:06:41.215801 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:06:41.215803 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:06:41.216513 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:06:41.216514 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:06:41.217089 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:06:41.217091 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:06:41.217460 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:06:41.217461 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:06:41.217524 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:06:41.217545 (+ 21us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 04:06:41.217840 (+ 295us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 04:06:41.217851 (+ 11us) client_negotiation.cc:769] Sending connection context
0810 04:06:41.217899 (+ 48us) client_negotiation.cc:240] Negotiation successful
0810 04:06:41.217944 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126}
I20260810 04:06:41.217999 1205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:41.215399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41100 (local address 127.31.32.2:40031)
0810 04:06:41.215467 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0810 04:06:41.215469 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:06:41.215558 (+ 89us) server_negotiation.cc:407] Connection header received
0810 04:06:41.215644 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:06:41.215645 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:06:41.215663 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:06:41.215738 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 04:06:41.215866 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:41.216430 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:41.217177 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:41.217383 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:41.217459 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:06:41.217646 (+ 187us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 04:06:41.217757 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 04:06:41.217863 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0810 04:06:41.217905 (+ 42us) server_negotiation.cc:299] Negotiation successful
0810 04:06:41.217926 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260810 04:06:41.219116 818 raft_consensus.cc:1261] T f8c964ce460a4cbba491d8cd72cbc65e P 409400b2d53242c699ea0373947ebabd [term 1 FOLLOWER]: Refusing update from remote peer 0587f4699f2640d1aa4b3035bcb0da9b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:06:41.219223 1131 raft_consensus.cc:1261] T f8c964ce460a4cbba491d8cd72cbc65e P f12f746012f14fa983ea93b29069958e [term 1 FOLLOWER]: Refusing update from remote peer 0587f4699f2640d1aa4b3035bcb0da9b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:06:41.219480 1215 consensus_queue.cc:1035] T f8c964ce460a4cbba491d8cd72cbc65e P 0587f4699f2640d1aa4b3035bcb0da9b [LEADER]: Connected to new peer: Peer: permanent_uuid: "409400b2d53242c699ea0373947ebabd" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 36135 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:06:41.219614 1238 consensus_queue.cc:1035] T f8c964ce460a4cbba491d8cd72cbc65e P 0587f4699f2640d1aa4b3035bcb0da9b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12f746012f14fa983ea93b29069958e" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 45985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:06:41.220192 1210 tablet_replica.cc:439] T f8c964ce460a4cbba491d8cd72cbc65e P 409400b2d53242c699ea0373947ebabd: TxnStatusTablet state changed. Reason: New leader 0587f4699f2640d1aa4b3035bcb0da9b. Latest consensus state: current_term: 1 leader_uuid: "0587f4699f2640d1aa4b3035bcb0da9b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "409400b2d53242c699ea0373947ebabd" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 36135 } } peers { permanent_uuid: "0587f4699f2640d1aa4b3035bcb0da9b" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 40031 } } peers { permanent_uuid: "f12f746012f14fa983ea93b29069958e" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 45985 } } }
I20260810 04:06:41.220305 1210 tablet_replica.cc:442] T f8c964ce460a4cbba491d8cd72cbc65e P 409400b2d53242c699ea0373947ebabd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 04:06:41.220307 1226 tablet_replica.cc:439] T f8c964ce460a4cbba491d8cd72cbc65e P f12f746012f14fa983ea93b29069958e: TxnStatusTablet state changed. Reason: New leader 0587f4699f2640d1aa4b3035bcb0da9b. Latest consensus state: current_term: 1 leader_uuid: "0587f4699f2640d1aa4b3035bcb0da9b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "409400b2d53242c699ea0373947ebabd" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 36135 } } peers { permanent_uuid: "0587f4699f2640d1aa4b3035bcb0da9b" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 40031 } } peers { permanent_uuid: "f12f746012f14fa983ea93b29069958e" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 45985 } } }
I20260810 04:06:41.220404 1226 tablet_replica.cc:442] T f8c964ce460a4cbba491d8cd72cbc65e P f12f746012f14fa983ea93b29069958e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 04:06:41.220988 1238 tablet_replica.cc:439] T f8c964ce460a4cbba491d8cd72cbc65e P 0587f4699f2640d1aa4b3035bcb0da9b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0587f4699f2640d1aa4b3035bcb0da9b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "409400b2d53242c699ea0373947ebabd" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 36135 } } peers { permanent_uuid: "0587f4699f2640d1aa4b3035bcb0da9b" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 40031 } } peers { permanent_uuid: "f12f746012f14fa983ea93b29069958e" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 45985 } } }
I20260810 04:06:41.221069 1238 tablet_replica.cc:442] T f8c964ce460a4cbba491d8cd72cbc65e P 0587f4699f2640d1aa4b3035bcb0da9b: This TxnStatusTablet replica's current role is: LEADER
I20260810 04:06:41.221215 1222 mvcc.cc:205] Tried to move back new op lower bound from 7316827345791483904 to 7316827345624694784. Current Snapshot: MvccSnapshot[applied={T|T < 7316827345791483904 or (T in {7316827345791483904})}]
I20260810 04:06:41.221460 1215 tablet_replica.cc:439] T f8c964ce460a4cbba491d8cd72cbc65e P 0587f4699f2640d1aa4b3035bcb0da9b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0587f4699f2640d1aa4b3035bcb0da9b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "409400b2d53242c699ea0373947ebabd" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 36135 } } peers { permanent_uuid: "0587f4699f2640d1aa4b3035bcb0da9b" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 40031 } } peers { permanent_uuid: "f12f746012f14fa983ea93b29069958e" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 45985 } } }
I20260810 04:06:41.221545 1215 tablet_replica.cc:442] T f8c964ce460a4cbba491d8cd72cbc65e P 0587f4699f2640d1aa4b3035bcb0da9b: This TxnStatusTablet replica's current role is: LEADER
I20260810 04:06:41.221513 1221 mvcc.cc:205] Tried to move back new op lower bound from 7316827345791483904 to 7316827345624694784. Current Snapshot: MvccSnapshot[applied={T|T < 7316827345791483904}]
I20260810 04:06:41.221832 1226 tablet_replica.cc:439] T f8c964ce460a4cbba491d8cd72cbc65e P f12f746012f14fa983ea93b29069958e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0587f4699f2640d1aa4b3035bcb0da9b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "409400b2d53242c699ea0373947ebabd" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 36135 } } peers { permanent_uuid: "0587f4699f2640d1aa4b3035bcb0da9b" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 40031 } } peers { permanent_uuid: "f12f746012f14fa983ea93b29069958e" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 45985 } } }
I20260810 04:06:41.221903 1226 tablet_replica.cc:442] T f8c964ce460a4cbba491d8cd72cbc65e P f12f746012f14fa983ea93b29069958e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 04:06:41.222653 1210 tablet_replica.cc:439] T f8c964ce460a4cbba491d8cd72cbc65e P 409400b2d53242c699ea0373947ebabd: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0587f4699f2640d1aa4b3035bcb0da9b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "409400b2d53242c699ea0373947ebabd" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 36135 } } peers { permanent_uuid: "0587f4699f2640d1aa4b3035bcb0da9b" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 40031 } } peers { permanent_uuid: "f12f746012f14fa983ea93b29069958e" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 45985 } } }
I20260810 04:06:41.222750 1210 tablet_replica.cc:442] T f8c964ce460a4cbba491d8cd72cbc65e P 409400b2d53242c699ea0373947ebabd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 04:06:41.226068 1205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:41.223521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41116 (local address 127.31.32.2:40031)
0810 04:06:41.223567 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0810 04:06:41.223568 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 04:06:41.223615 (+ 47us) server_negotiation.cc:407] Connection header received
0810 04:06:41.223709 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:06:41.223710 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:06:41.223731 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:06:41.223792 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 04:06:41.223899 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:41.224509 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:41.225538 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:41.225698 (+ 160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:41.225744 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:06:41.225890 (+ 146us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 04:06:41.225911 (+ 21us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 04:06:41.225953 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0810 04:06:41.226013 (+ 60us) server_negotiation.cc:299] Negotiation successful
0810 04:06:41.226026 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260810 04:06:41.230923 1205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:41.228478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.2:46253 (local address 127.31.32.2:40031)
0810 04:06:41.228512 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0810 04:06:41.228514 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:06:41.228665 (+ 151us) server_negotiation.cc:407] Connection header received
0810 04:06:41.228710 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:06:41.228710 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:06:41.228717 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:06:41.228760 (+ 43us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 04:06:41.228884 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:41.229288 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:41.230352 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:41.230510 (+ 158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:41.230554 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:06:41.230737 (+ 183us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 04:06:41.230757 (+ 20us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 04:06:41.230793 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0810 04:06:41.230840 (+ 47us) server_negotiation.cc:299] Negotiation successful
0810 04:06:41.230853 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":18}
I20260810 04:06:41.231117 1263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:41.228414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.2:40031 (local address 127.31.32.2:46253)
0810 04:06:41.228590 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0810 04:06:41.228600 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 04:06:41.228658 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:06:41.228764 (+ 106us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:06:41.228765 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:06:41.228766 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 04:06:41.228818 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:06:41.228819 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:06:41.229374 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:06:41.229375 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:06:41.230272 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:06:41.230274 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:06:41.230577 (+ 303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:06:41.230578 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:06:41.230647 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:06:41.230658 (+ 11us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 04:06:41.230794 (+ 136us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 04:06:41.230795 (+ 1us) client_negotiation.cc:769] Sending connection context
0810 04:06:41.230834 (+ 39us) client_negotiation.cc:240] Negotiation successful
0810 04:06:41.230870 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":82,"threads_started":1}
I20260810 04:06:41.233451 1131 raft_consensus.cc:1261] T 34726e1ac2ac4c3f9374203d18ac729d P f12f746012f14fa983ea93b29069958e [term 1 FOLLOWER]: Refusing update from remote peer 0587f4699f2640d1aa4b3035bcb0da9b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:06:41.233456 818 raft_consensus.cc:1261] T 34726e1ac2ac4c3f9374203d18ac729d P 409400b2d53242c699ea0373947ebabd [term 1 FOLLOWER]: Refusing update from remote peer 0587f4699f2640d1aa4b3035bcb0da9b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:06:41.233649 1264 consensus_queue.cc:1035] T 34726e1ac2ac4c3f9374203d18ac729d P 0587f4699f2640d1aa4b3035bcb0da9b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12f746012f14fa983ea93b29069958e" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 45985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:06:41.233769 1238 consensus_queue.cc:1035] T 34726e1ac2ac4c3f9374203d18ac729d P 0587f4699f2640d1aa4b3035bcb0da9b [LEADER]: Connected to new peer: Peer: permanent_uuid: "409400b2d53242c699ea0373947ebabd" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 36135 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:06:41.245817 641 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33394:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 04:06:41.245952 641 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33394:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 04:06:41.246615 641 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b898b856782d4f13872e6e2373d5a953: Sending DeleteTablet for 3 replicas of tablet 34726e1ac2ac4c3f9374203d18ac729d
I20260810 04:06:41.246838 795 tablet_service.cc:1514] Processing DeleteTablet for tablet 34726e1ac2ac4c3f9374203d18ac729d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:06:41 UTC) from {username='kudu', principal='kudu/127.31.32.62@KRBTEST.COM'} at 127.0.0.1:35874
I20260810 04:06:41.246920 949 tablet_service.cc:1514] Processing DeleteTablet for tablet 34726e1ac2ac4c3f9374203d18ac729d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:06:41 UTC) from {username='kudu', principal='kudu/127.31.32.62@KRBTEST.COM'} at 127.0.0.1:41090
I20260810 04:06:41.246987 1231 tablet_replica.cc:330] T 34726e1ac2ac4c3f9374203d18ac729d P 409400b2d53242c699ea0373947ebabd: stopping tablet replica
I20260810 04:06:41.247043 1231 raft_consensus.cc:2229] T 34726e1ac2ac4c3f9374203d18ac729d P 409400b2d53242c699ea0373947ebabd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 04:06:41.247074 1231 raft_consensus.cc:2258] T 34726e1ac2ac4c3f9374203d18ac729d P 409400b2d53242c699ea0373947ebabd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:06:41.247146 1109 tablet_service.cc:1514] Processing DeleteTablet for tablet 34726e1ac2ac4c3f9374203d18ac729d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:06:41 UTC) from {username='kudu', principal='kudu/127.31.32.62@KRBTEST.COM'} at 127.0.0.1:35194
I20260810 04:06:41.247236 1232 tablet_replica.cc:330] T 34726e1ac2ac4c3f9374203d18ac729d P f12f746012f14fa983ea93b29069958e: stopping tablet replica
I20260810 04:06:41.247298 1232 raft_consensus.cc:2229] T 34726e1ac2ac4c3f9374203d18ac729d P f12f746012f14fa983ea93b29069958e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 04:06:41.247330 1232 raft_consensus.cc:2258] T 34726e1ac2ac4c3f9374203d18ac729d P f12f746012f14fa983ea93b29069958e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:06:41.247537 1231 ts_tablet_manager.cc:1905] T 34726e1ac2ac4c3f9374203d18ac729d P 409400b2d53242c699ea0373947ebabd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:06:41.247623 1230 tablet_replica.cc:330] T 34726e1ac2ac4c3f9374203d18ac729d P 0587f4699f2640d1aa4b3035bcb0da9b: stopping tablet replica
I20260810 04:06:41.247639 1232 ts_tablet_manager.cc:1905] T 34726e1ac2ac4c3f9374203d18ac729d P f12f746012f14fa983ea93b29069958e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:06:41.247674 1230 raft_consensus.cc:2229] T 34726e1ac2ac4c3f9374203d18ac729d P 0587f4699f2640d1aa4b3035bcb0da9b [term 1 LEADER]: Raft consensus shutting down.
I20260810 04:06:41.247785 1230 raft_consensus.cc:2258] T 34726e1ac2ac4c3f9374203d18ac729d P 0587f4699f2640d1aa4b3035bcb0da9b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:06:41.248087 1230 ts_tablet_manager.cc:1905] T 34726e1ac2ac4c3f9374203d18ac729d P 0587f4699f2640d1aa4b3035bcb0da9b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:06:41.249208 1232 ts_tablet_manager.cc:1918] T 34726e1ac2ac4c3f9374203d18ac729d P f12f746012f14fa983ea93b29069958e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 04:06:41.249265 1232 log.cc:1196] T 34726e1ac2ac4c3f9374203d18ac729d P f12f746012f14fa983ea93b29069958e: Deleting WAL directory at /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786334794676589-31872-0/minicluster-data/ts-2/wal/wals/34726e1ac2ac4c3f9374203d18ac729d
I20260810 04:06:41.249302 1231 ts_tablet_manager.cc:1918] T 34726e1ac2ac4c3f9374203d18ac729d P 409400b2d53242c699ea0373947ebabd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 04:06:41.249354 1231 log.cc:1196] T 34726e1ac2ac4c3f9374203d18ac729d P 409400b2d53242c699ea0373947ebabd: Deleting WAL directory at /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786334794676589-31872-0/minicluster-data/ts-0/wal/wals/34726e1ac2ac4c3f9374203d18ac729d
I20260810 04:06:41.249490 1232 ts_tablet_manager.cc:1939] T 34726e1ac2ac4c3f9374203d18ac729d P f12f746012f14fa983ea93b29069958e: Deleting consensus metadata
I20260810 04:06:41.249583 1231 ts_tablet_manager.cc:1939] T 34726e1ac2ac4c3f9374203d18ac729d P 409400b2d53242c699ea0373947ebabd: Deleting consensus metadata
I20260810 04:06:41.249876 1230 ts_tablet_manager.cc:1918] T 34726e1ac2ac4c3f9374203d18ac729d P 0587f4699f2640d1aa4b3035bcb0da9b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 04:06:41.249928 1230 log.cc:1196] T 34726e1ac2ac4c3f9374203d18ac729d P 0587f4699f2640d1aa4b3035bcb0da9b: Deleting WAL directory at /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786334794676589-31872-0/minicluster-data/ts-1/wal/wals/34726e1ac2ac4c3f9374203d18ac729d
I20260810 04:06:41.250025 624 catalog_manager.cc:4841] TS 409400b2d53242c699ea0373947ebabd (127.31.32.1:36135): tablet 34726e1ac2ac4c3f9374203d18ac729d (table test-table [id=d68264b483de4a8aba2b17ab38f2012a]) successfully deleted
I20260810 04:06:41.250154 1230 ts_tablet_manager.cc:1939] T 34726e1ac2ac4c3f9374203d18ac729d P 0587f4699f2640d1aa4b3035bcb0da9b: Deleting consensus metadata
I20260810 04:06:41.250468 620 catalog_manager.cc:4841] TS 0587f4699f2640d1aa4b3035bcb0da9b (127.31.32.2:40031): tablet 34726e1ac2ac4c3f9374203d18ac729d (table test-table [id=d68264b483de4a8aba2b17ab38f2012a]) successfully deleted
I20260810 04:06:41.250588 624 catalog_manager.cc:4841] TS f12f746012f14fa983ea93b29069958e (127.31.32.3:45985): tablet 34726e1ac2ac4c3f9374203d18ac729d (table test-table [id=d68264b483de4a8aba2b17ab38f2012a]) successfully deleted
Aug 10 04:06:41 dist-test-slave-8w9v krb5kdc[589](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334800, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.31.32.1@KRBTEST.COM
I20260810 04:06:41.259347 1201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:41.249913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35878 (local address 127.31.32.1:36135)
0810 04:06:41.249965 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0810 04:06:41.249967 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:06:41.249979 (+ 12us) server_negotiation.cc:407] Connection header received
0810 04:06:41.250116 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:06:41.250117 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:06:41.250130 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:06:41.250157 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:06:41.251004 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:41.251697 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:41.252688 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:41.252977 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:41.253059 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:06:41.256291 (+ 3232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:06:41.256301 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:06:41.256301 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:06:41.256309 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:06:41.258175 (+ 1866us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:06:41.258532 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:06:41.258536 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:06:41.258537 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:06:41.258605 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:06:41.258795 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:06:41.258796 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:06:41.258797 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:06:41.258975 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:06:41.259134 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0810 04:06:41.259174 (+ 40us) server_negotiation.cc:299] Negotiation successful
0810 04:06:41.259258 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
W20260810 04:06:41.259663 864 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35878
I20260810 04:06:41.267016 696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:41.260757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33414 (local address 127.31.32.62:38035)
0810 04:06:41.260805 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0810 04:06:41.260807 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:06:41.260872 (+ 65us) server_negotiation.cc:407] Connection header received
0810 04:06:41.260937 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:06:41.260938 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:06:41.260950 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:06:41.261011 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:06:41.261463 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:41.262205 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:41.262872 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:41.263170 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:41.263244 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:06:41.264446 (+ 1202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:06:41.264448 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:06:41.264448 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:06:41.264456 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:06:41.265958 (+ 1502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:06:41.266279 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:06:41.266280 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:06:41.266281 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:06:41.266346 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:06:41.266528 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:06:41.266529 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:06:41.266529 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:06:41.266663 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:06:41.266754 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0810 04:06:41.266851 (+ 97us) server_negotiation.cc:299] Negotiation successful
0810 04:06:41.266930 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
W20260810 04:06:41.267251 641 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33414
I20260810 04:06:41.267935 31872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskws89Io/build/release/bin/kudu with pid 719
I20260810 04:06:41.281543 31872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskws89Io/build/release/bin/kudu with pid 879
I20260810 04:06:41.289608 31872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskws89Io/build/release/bin/kudu with pid 1034
I20260810 04:06:41.297067 31872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskws89Io/build/release/bin/kudu with pid 603
2026-08-10T04:06:41Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2285 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786334794676589-31872-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 04:06:41 dist-test-slave-8w9v krb5kdc[1277](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 04:06:41 dist-test-slave-8w9v krb5kdc[1277](info): set up 2 sockets
Aug 10 04:06:41 dist-test-slave-8w9v krb5kdc[1277](info): commencing operation
krb5kdc: starting...
W20260810 04:06:42.371213 31872 mini_kdc.cc:121] Time spent starting KDC: real 1.051s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786334794676589-31872-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786334794676589-31872-0/krb5kdc/test-user.keytab.
Aug 10 04:06:42 dist-test-slave-8w9v krb5kdc[1277](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334802, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T04:06:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T04:06:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.32.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.32.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.32.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.62.keytab.
Entry for principal kudu/127.31.32.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.62.keytab.
I20260810 04:06:42.491721 31872 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskws89Io/build/release/bin/kudu
/tmp/dist-test-taskws89Io/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786334794676589-31872-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786334794676589-31872-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786334794676589-31872-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786334794676589-31872-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.32.62:34751
--webserver_interface=127.31.32.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.62.keytab
--principal=kudu/127.31.32.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.32.20:41897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.32.62:34751
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786334794676589-31872-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786334794676589-31872-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786334794676589-31872-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:06:42.573938 1292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:06:42.574180 1292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:06:42.574218 1292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:06:42.575378 1292 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:06:42.575448 1292 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260810 04:06:42.575469 1292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:06:42.575506 1292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:06:42.575539 1292 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:06:42.575565 1292 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:06:42.577119 1292 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.32.20:41897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786334794676589-31872-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786334794676589-31872-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.32.62:34751
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.62.keytab
--principal=kudu/127.31.32.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.32.62:34751
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786334794676589-31872-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.32.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786334794676589-31872-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:06:42.577380 1292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:06:42.577613 1292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:06:42.582392 1301 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:06:42.582436 1298 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:06:42.582386 1299 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:06:42.582715 1292 server_base.cc:1034] running on GCE node
I20260810 04:06:42.583001 1292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:06:42.583280 1292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:06:42.584415 1292 hybrid_clock.cc:648] HybridClock initialized: now 1786334802584402 us; error 26 us; skew 500 ppm
Aug 10 04:06:42 dist-test-slave-8w9v krb5kdc[1277](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334802, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:06:42.586747 1292 init.cc:377] Logged in from keytab as kudu/127.31.32.62@KRBTEST.COM (short username kudu)
I20260810 04:06:42.587208 1292 webserver.cc:460] Webserver started at http://127.31.32.62:38003/ using document root <none> and password file <none>
I20260810 04:06:42.587401 1292 fs_manager.cc:359] Metadata directory not provided
I20260810 04:06:42.587445 1292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:06:42.587534 1292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:06:42.588549 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786334794676589-31872-0/minicluster-data/master-0/data/instance:
uuid: "6fa0f90143354e5bb2c4cbac1d334676"
format_stamp: "Formatted at 2026-08-10 04:06:42 on dist-test-slave-8w9v"
server_key: "82ca868160d4cf60c149adb6586d615b"
server_key_iv: "fac5e3c416f4084b0dc34bd49d982bae"
server_key_version: "encryptionkey@0"
I20260810 04:06:42.588860 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786334794676589-31872-0/minicluster-data/master-0/wal/instance:
uuid: "6fa0f90143354e5bb2c4cbac1d334676"
format_stamp: "Formatted at 2026-08-10 04:06:42 on dist-test-slave-8w9v"
server_key: "82ca868160d4cf60c149adb6586d615b"
server_key_iv: "fac5e3c416f4084b0dc34bd49d982bae"
server_key_version: "encryptionkey@0"
I20260810 04:06:42.590065 1292 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 04:06:42.590739 1308 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:06:42.590919 1292 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 04:06:42.591024 1292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786334794676589-31872-0/minicluster-data/master-0/data,/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786334794676589-31872-0/minicluster-data/master-0/wal
uuid: "6fa0f90143354e5bb2c4cbac1d334676"
format_stamp: "Formatted at 2026-08-10 04:06:42 on dist-test-slave-8w9v"
server_key: "82ca868160d4cf60c149adb6586d615b"
server_key_iv: "fac5e3c416f4084b0dc34bd49d982bae"
server_key_version: "encryptionkey@0"
I20260810 04:06:42.591099 1292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786334794676589-31872-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786334794676589-31872-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786334794676589-31872-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:06:42.608995 1292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:06:42.610705 1292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:06:42.610872 1292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:06:42.615547 1292 rpc_server.cc:307] RPC server started. Bound to: 127.31.32.62:34751
I20260810 04:06:42.615600 1377 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.32.62:34751 every 8 connection(s)
I20260810 04:06:42.615908 1292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786334794676589-31872-0/minicluster-data/master-0/data/info.pb
I20260810 04:06:42.616588 1378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:06:42.616757 31872 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskws89Io/build/release/bin/kudu as pid 1292
I20260810 04:06:42.616849 31872 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786334794676589-31872-0/minicluster-data/master-0/wal/instance
I20260810 04:06:42.617027 31872 external_mini_cluster.cc:1428] Setting key a8e0acab4afee54aeb63879c72474b71
I20260810 04:06:42.619998 1378 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6fa0f90143354e5bb2c4cbac1d334676: Bootstrap starting.
I20260810 04:06:42.620728 1378 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6fa0f90143354e5bb2c4cbac1d334676: Neither blocks nor log segments found. Creating new log.
I20260810 04:06:42.621030 1378 log.cc:824] T 00000000000000000000000000000000 P 6fa0f90143354e5bb2c4cbac1d334676: Log is configured to *not* fsync() on all Append() calls
I20260810 04:06:42.621768 1378 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6fa0f90143354e5bb2c4cbac1d334676: No bootstrap required, opened a new log
Aug 10 04:06:42 dist-test-slave-8w9v krb5kdc[1277](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334802, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.32.62@KRBTEST.COM
I20260810 04:06:42.623459 1378 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6fa0f90143354e5bb2c4cbac1d334676 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fa0f90143354e5bb2c4cbac1d334676" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 34751 } }
I20260810 04:06:42.623621 1378 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6fa0f90143354e5bb2c4cbac1d334676 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:06:42.623658 1378 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6fa0f90143354e5bb2c4cbac1d334676 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6fa0f90143354e5bb2c4cbac1d334676, State: Initialized, Role: FOLLOWER
I20260810 04:06:42.623776 1378 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6fa0f90143354e5bb2c4cbac1d334676 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fa0f90143354e5bb2c4cbac1d334676" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 34751 } }
I20260810 04:06:42.623847 1378 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6fa0f90143354e5bb2c4cbac1d334676 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 04:06:42.623868 1378 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6fa0f90143354e5bb2c4cbac1d334676 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 04:06:42.623934 1378 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6fa0f90143354e5bb2c4cbac1d334676 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:06:42.624728 1378 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6fa0f90143354e5bb2c4cbac1d334676 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fa0f90143354e5bb2c4cbac1d334676" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 34751 } }
I20260810 04:06:42.624845 1378 leader_election.cc:302] T 00000000000000000000000000000000 P 6fa0f90143354e5bb2c4cbac1d334676 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6fa0f90143354e5bb2c4cbac1d334676; no voters:
I20260810 04:06:42.625015 1378 leader_election.cc:288] T 00000000000000000000000000000000 P 6fa0f90143354e5bb2c4cbac1d334676 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 04:06:42.625074 1384 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6fa0f90143354e5bb2c4cbac1d334676 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:06:42.625222 1384 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6fa0f90143354e5bb2c4cbac1d334676 [term 1 LEADER]: Becoming Leader. State: Replica: 6fa0f90143354e5bb2c4cbac1d334676, State: Running, Role: LEADER
I20260810 04:06:42.625280 1378 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6fa0f90143354e5bb2c4cbac1d334676 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:06:42.625311 1384 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6fa0f90143354e5bb2c4cbac1d334676 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fa0f90143354e5bb2c4cbac1d334676" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 34751 } }
I20260810 04:06:42.625594 1384 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fa0f90143354e5bb2c4cbac1d334676 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6fa0f90143354e5bb2c4cbac1d334676. Latest consensus state: current_term: 1 leader_uuid: "6fa0f90143354e5bb2c4cbac1d334676" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fa0f90143354e5bb2c4cbac1d334676" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 34751 } } }
I20260810 04:06:42.625681 1384 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fa0f90143354e5bb2c4cbac1d334676 [sys.catalog]: This master's current role is: LEADER
I20260810 04:06:42.625603 1385 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fa0f90143354e5bb2c4cbac1d334676 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6fa0f90143354e5bb2c4cbac1d334676" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fa0f90143354e5bb2c4cbac1d334676" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 34751 } } }
I20260810 04:06:42.625818 1385 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fa0f90143354e5bb2c4cbac1d334676 [sys.catalog]: This master's current role is: LEADER
W20260810 04:06:42.627090 1399 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6fa0f90143354e5bb2c4cbac1d334676: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 04:06:42.627151 1399 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 04:06:42.627465 1390 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 04:06:42.627635 1390 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 04:06:42.627882 1382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:42.617535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44222 (local address 127.31.32.62:34751)
0810 04:06:42.617729 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0810 04:06:42.617738 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 04:06:42.617771 (+ 33us) server_negotiation.cc:407] Connection header received
0810 04:06:42.618268 (+ 497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:06:42.618278 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:06:42.618339 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:06:42.618486 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:06:42.619042 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:42.620413 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:42.621194 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:42.621696 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:42.621806 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:06:42.624076 (+ 2270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:06:42.624080 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:06:42.624080 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:06:42.624092 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:06:42.626143 (+ 2051us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:06:42.626459 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:06:42.626462 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:06:42.626463 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:06:42.626578 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:06:42.626827 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:06:42.626828 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:06:42.626829 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:06:42.627137 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:06:42.627303 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0810 04:06:42.627557 (+ 254us) server_negotiation.cc:299] Negotiation successful
0810 04:06:42.627691 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":87,"threads_started":1}
I20260810 04:06:42.629113 1390 catalog_manager.cc:1347] Generated new cluster ID: c0cd538f8a0a4efb96a975973fc743ef
I20260810 04:06:42.629164 1390 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 04:06:42.647908 1390 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 04:06:42.648502 1390 catalog_manager.cc:1504] Loading token signing keys...
I20260810 04:06:42.655859 1390 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6fa0f90143354e5bb2c4cbac1d334676: Generated new TSK 0
I20260810 04:06:42.656097 1390 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.32.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.32.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.32.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.1.keytab.
Entry for principal kudu/127.31.32.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.1.keytab.
I20260810 04:06:42.688637 31872 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskws89Io/build/release/bin/kudu
/tmp/dist-test-taskws89Io/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786334794676589-31872-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786334794676589-31872-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786334794676589-31872-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786334794676589-31872-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.32.1:0
--local_ip_for_outbound_sockets=127.31.32.1
--webserver_interface=127.31.32.1
--webserver_port=0
--tserver_master_addrs=127.31.32.62:34751
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.1.keytab
--principal=kudu/127.31.32.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.32.20:41897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786334794676589-31872-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786334794676589-31872-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786334794676589-31872-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:06:42.768671 1406 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:06:42.768850 1406 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:06:42.768874 1406 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:06:42.769963 1406 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:06:42.770002 1406 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:06:42.770022 1406 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260810 04:06:42.770035 1406 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 04:06:42.770072 1406 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.32.1
I20260810 04:06:42.771788 1406 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.32.20:41897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786334794676589-31872-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786334794676589-31872-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.1.keytab
--principal=kudu/127.31.32.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.32.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786334794676589-31872-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.32.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.31.32.62:34751
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1406
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.32.1
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786334794676589-31872-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:06:42.772045 1406 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:06:42.772332 1406 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:06:42.777455 1412 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:06:42.777455 1415 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:06:42.777449 1411 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:06:42.777743 1406 server_base.cc:1034] running on GCE node
I20260810 04:06:42.777918 1406 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:06:42.778155 1406 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:06:42.779312 1406 hybrid_clock.cc:648] HybridClock initialized: now 1786334802779300 us; error 28 us; skew 500 ppm
Aug 10 04:06:42 dist-test-slave-8w9v krb5kdc[1277](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334802, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:06:42.781824 1406 init.cc:377] Logged in from keytab as kudu/127.31.32.1@KRBTEST.COM (short username kudu)
I20260810 04:06:42.782269 1406 webserver.cc:460] Webserver started at http://127.31.32.1:38339/ using document root <none> and password file <none>
I20260810 04:06:42.782470 1406 fs_manager.cc:359] Metadata directory not provided
I20260810 04:06:42.782509 1406 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:06:42.782593 1406 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:06:42.783519 1406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786334794676589-31872-0/minicluster-data/ts-0/data/instance:
uuid: "3fd00917592c4e1e91b1e32810acd65a"
format_stamp: "Formatted at 2026-08-10 04:06:42 on dist-test-slave-8w9v"
server_key: "fa08bdd6f07d1f53ece88cded0ee58a9"
server_key_iv: "ead6272b9d99d011732cb6c46f330eb5"
server_key_version: "encryptionkey@0"
I20260810 04:06:42.783805 1406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786334794676589-31872-0/minicluster-data/ts-0/wal/instance:
uuid: "3fd00917592c4e1e91b1e32810acd65a"
format_stamp: "Formatted at 2026-08-10 04:06:42 on dist-test-slave-8w9v"
server_key: "fa08bdd6f07d1f53ece88cded0ee58a9"
server_key_iv: "ead6272b9d99d011732cb6c46f330eb5"
server_key_version: "encryptionkey@0"
I20260810 04:06:42.785146 1406 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 04:06:42.785869 1422 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:06:42.786048 1406 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 04:06:42.786163 1406 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786334794676589-31872-0/minicluster-data/ts-0/data,/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786334794676589-31872-0/minicluster-data/ts-0/wal
uuid: "3fd00917592c4e1e91b1e32810acd65a"
format_stamp: "Formatted at 2026-08-10 04:06:42 on dist-test-slave-8w9v"
server_key: "fa08bdd6f07d1f53ece88cded0ee58a9"
server_key_iv: "ead6272b9d99d011732cb6c46f330eb5"
server_key_version: "encryptionkey@0"
I20260810 04:06:42.786237 1406 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786334794676589-31872-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786334794676589-31872-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786334794676589-31872-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:06:42.819182 1406 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:06:42.821653 1406 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:06:42.821833 1406 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:06:42.822417 1406 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:06:42.822474 1406 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:06:42.822513 1406 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:06:42.822541 1406 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 04:06:42 dist-test-slave-8w9v krb5kdc[1277](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334802, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.1@KRBTEST.COM for kudu/127.31.32.62@KRBTEST.COM
I20260810 04:06:42.829938 1406 rpc_server.cc:307] RPC server started. Bound to: 127.31.32.1:33943
I20260810 04:06:42.830268 1406 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786334794676589-31872-0/minicluster-data/ts-0/data/info.pb
I20260810 04:06:42.830771 1548 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.32.1:33943 every 8 connection(s)
I20260810 04:06:42.832763 1382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:42.822895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.1:41383 (local address 127.31.32.62:34751)
0810 04:06:42.822961 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0810 04:06:42.822963 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:06:42.823171 (+ 208us) server_negotiation.cc:407] Connection header received
0810 04:06:42.823760 (+ 589us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:06:42.823762 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:06:42.823776 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:06:42.823848 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:06:42.824565 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:42.825091 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:42.826854 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:42.827135 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:42.827238 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:06:42.829974 (+ 2736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:06:42.829976 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:06:42.829976 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:06:42.829985 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:06:42.831545 (+ 1560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:06:42.831919 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:06:42.831920 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:06:42.831920 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:06:42.831977 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:06:42.832236 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:06:42.832236 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:06:42.832237 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:06:42.832347 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:06:42.832410 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0810 04:06:42.832620 (+ 210us) server_negotiation.cc:299] Negotiation successful
0810 04:06:42.832691 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260810 04:06:42.832901 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:42.822861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:34751 (local address 127.31.32.1:41383)
0810 04:06:42.823030 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0810 04:06:42.823065 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0810 04:06:42.823649 (+ 584us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:06:42.823960 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:06:42.823964 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:06:42.824260 (+ 296us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:06:42.824451 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:06:42.824456 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:06:42.825208 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:06:42.825210 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:06:42.826721 (+ 1511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:06:42.826725 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:06:42.827272 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:06:42.827274 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:06:42.827412 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:06:42.827943 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:06:42.827953 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:06:42.829857 (+ 1904us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:06:42.831621 (+ 1764us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:06:42.831627 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:06:42.831628 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:06:42.831848 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:06:42.832037 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:06:42.832037 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:06:42.832038 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:06:42.832179 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:06:42.832402 (+ 223us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:06:42.832405 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:06:42.832506 (+ 101us) client_negotiation.cc:769] Sending connection context
0810 04:06:42.832575 (+ 69us) client_negotiation.cc:240] Negotiation successful
0810 04:06:42.832687 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":90,"threads_started":1}
I20260810 04:06:42.833427 1549 heartbeater.cc:347] Connected to a master server at 127.31.32.62:34751
I20260810 04:06:42.833521 1549 heartbeater.cc:464] Registering TS with master...
I20260810 04:06:42.833617 31872 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskws89Io/build/release/bin/kudu as pid 1406
I20260810 04:06:42.833710 1549 heartbeater.cc:511] Master 127.31.32.62:34751 requested a full tablet report, sending...
I20260810 04:06:42.833720 31872 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786334794676589-31872-0/minicluster-data/ts-0/wal/instance
I20260810 04:06:42.833937 31872 external_mini_cluster.cc:1428] Setting key d02297fcda573579c6c2a6f4fac47283
I20260810 04:06:42.834324 1325 ts_manager.cc:194] Registered new tserver with Master: 3fd00917592c4e1e91b1e32810acd65a (127.31.32.1:33943)
I20260810 04:06:42.835173 1325 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.32.1@KRBTEST.COM'} at 127.31.32.1:41383
I20260810 04:06:42.840781 1382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:42.834593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.1:36973 (local address 127.31.32.62:34751)
0810 04:06:42.834637 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0810 04:06:42.834639 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:06:42.834758 (+ 119us) server_negotiation.cc:407] Connection header received
0810 04:06:42.834814 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:06:42.834815 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:06:42.834826 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:06:42.834895 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:06:42.835292 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:42.835867 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:42.836717 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:42.836919 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:42.837009 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:06:42.838050 (+ 1041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:06:42.838052 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:06:42.838052 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:06:42.838059 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:06:42.839542 (+ 1483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:06:42.839967 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:06:42.839968 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:06:42.839968 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:06:42.840027 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:06:42.840252 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:06:42.840253 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:06:42.840253 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:06:42.840375 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:06:42.840446 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0810 04:06:42.840618 (+ 172us) server_negotiation.cc:299] Negotiation successful
0810 04:06:42.840693 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260810 04:06:42.840796 1555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:42.834548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:34751 (local address 127.31.32.1:36973)
0810 04:06:42.834694 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0810 04:06:42.834705 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 04:06:42.834765 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:06:42.834902 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:06:42.834903 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:06:42.835125 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:06:42.835198 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:06:42.835200 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:06:42.835972 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:06:42.835973 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:06:42.836605 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:06:42.836608 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:06:42.837011 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:06:42.837012 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:06:42.837096 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:06:42.837597 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:06:42.837602 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:06:42.837933 (+ 331us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:06:42.839695 (+ 1762us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:06:42.839696 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:06:42.839696 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:06:42.839880 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:06:42.840099 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:06:42.840099 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:06:42.840100 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:06:42.840180 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:06:42.840449 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:06:42.840450 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:06:42.840505 (+ 55us) client_negotiation.cc:769] Sending connection context
0810 04:06:42.840592 (+ 87us) client_negotiation.cc:240] Negotiation successful
0810 04:06:42.840696 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":76,"threads_started":1}
WARNING: no policy specified for kudu/127.31.32.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.32.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.32.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.2.keytab.
Entry for principal kudu/127.31.32.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.2.keytab.
I20260810 04:06:42.860648 31872 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskws89Io/build/release/bin/kudu
/tmp/dist-test-taskws89Io/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786334794676589-31872-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786334794676589-31872-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786334794676589-31872-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786334794676589-31872-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.32.2:0
--local_ip_for_outbound_sockets=127.31.32.2
--webserver_interface=127.31.32.2
--webserver_port=0
--tserver_master_addrs=127.31.32.62:34751
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.2.keytab
--principal=kudu/127.31.32.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.32.20:41897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786334794676589-31872-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786334794676589-31872-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786334794676589-31872-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:06:42.940430 1558 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:06:42.940610 1558 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:06:42.940634 1558 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:06:42.941721 1558 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:06:42.941761 1558 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:06:42.941781 1558 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260810 04:06:42.941792 1558 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 04:06:42.941809 1558 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.32.2
I20260810 04:06:42.943171 1558 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.32.20:41897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786334794676589-31872-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786334794676589-31872-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.2.keytab
--principal=kudu/127.31.32.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.32.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786334794676589-31872-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.32.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.31.32.62:34751
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1558
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.32.2
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786334794676589-31872-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:06:42.943414 1558 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:06:42.943630 1558 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:06:42.948403 1568 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:06:42.948427 1566 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:06:42.948498 1565 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:06:42.948598 1558 server_base.cc:1034] running on GCE node
I20260810 04:06:42.948789 1558 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:06:42.948976 1558 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:06:42.950122 1558 hybrid_clock.cc:648] HybridClock initialized: now 1786334802950112 us; error 29 us; skew 500 ppm
Aug 10 04:06:42 dist-test-slave-8w9v krb5kdc[1277](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334802, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:06:42.952476 1558 init.cc:377] Logged in from keytab as kudu/127.31.32.2@KRBTEST.COM (short username kudu)
I20260810 04:06:42.952909 1558 webserver.cc:460] Webserver started at http://127.31.32.2:35089/ using document root <none> and password file <none>
I20260810 04:06:42.953099 1558 fs_manager.cc:359] Metadata directory not provided
I20260810 04:06:42.953136 1558 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:06:42.953230 1558 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:06:42.954165 1558 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786334794676589-31872-0/minicluster-data/ts-1/data/instance:
uuid: "554b70db09f344a2b8056e8d8bd6bdad"
format_stamp: "Formatted at 2026-08-10 04:06:42 on dist-test-slave-8w9v"
server_key: "fc997f176536059b843f356d8a929415"
server_key_iv: "37dcf6f4683754fa72d7d0968c39ba02"
server_key_version: "encryptionkey@0"
I20260810 04:06:42.954509 1558 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786334794676589-31872-0/minicluster-data/ts-1/wal/instance:
uuid: "554b70db09f344a2b8056e8d8bd6bdad"
format_stamp: "Formatted at 2026-08-10 04:06:42 on dist-test-slave-8w9v"
server_key: "fc997f176536059b843f356d8a929415"
server_key_iv: "37dcf6f4683754fa72d7d0968c39ba02"
server_key_version: "encryptionkey@0"
I20260810 04:06:42.955753 1558 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 04:06:42.956509 1576 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:06:42.956707 1558 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260810 04:06:42.956795 1558 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786334794676589-31872-0/minicluster-data/ts-1/data,/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786334794676589-31872-0/minicluster-data/ts-1/wal
uuid: "554b70db09f344a2b8056e8d8bd6bdad"
format_stamp: "Formatted at 2026-08-10 04:06:42 on dist-test-slave-8w9v"
server_key: "fc997f176536059b843f356d8a929415"
server_key_iv: "37dcf6f4683754fa72d7d0968c39ba02"
server_key_version: "encryptionkey@0"
I20260810 04:06:42.956853 1558 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786334794676589-31872-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786334794676589-31872-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786334794676589-31872-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:06:42.968626 1558 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:06:42.970865 1558 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:06:42.970975 1558 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:06:42.971429 1558 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:06:42.971467 1558 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:06:42.971493 1558 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:06:42.971508 1558 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:06:42.978806 1558 rpc_server.cc:307] RPC server started. Bound to: 127.31.32.2:35599
I20260810 04:06:42.978874 1697 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.32.2:35599 every 8 connection(s)
Aug 10 04:06:42 dist-test-slave-8w9v krb5kdc[1277](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334802, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.2@KRBTEST.COM for kudu/127.31.32.62@KRBTEST.COM
I20260810 04:06:42.979245 1558 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786334794676589-31872-0/minicluster-data/ts-1/data/info.pb
I20260810 04:06:42.982125 1382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:42.971994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.2:38781 (local address 127.31.32.62:34751)
0810 04:06:42.972072 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0810 04:06:42.972075 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:06:42.972278 (+ 203us) server_negotiation.cc:407] Connection header received
0810 04:06:42.972789 (+ 511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:06:42.972790 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:06:42.972803 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:06:42.972864 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:06:42.973804 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:42.974598 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:42.976494 (+ 1896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:42.976815 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:42.976873 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:06:42.979710 (+ 2837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:06:42.979712 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:06:42.979713 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:06:42.979721 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:06:42.980955 (+ 1234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:06:42.981352 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:06:42.981353 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:06:42.981353 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:06:42.981403 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:06:42.981606 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:06:42.981606 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:06:42.981607 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:06:42.981702 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:06:42.981751 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0810 04:06:42.981987 (+ 236us) server_negotiation.cc:299] Negotiation successful
0810 04:06:42.982063 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 04:06:42.982293 1587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:42.971959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:34751 (local address 127.31.32.2:38781)
0810 04:06:42.972163 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0810 04:06:42.972202 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0810 04:06:42.972678 (+ 476us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:06:42.972981 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:06:42.972986 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:06:42.973384 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:06:42.973692 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:06:42.973699 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:06:42.974752 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:06:42.974753 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:06:42.976382 (+ 1629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:06:42.976386 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:06:42.976897 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:06:42.976898 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:06:42.977059 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:06:42.977650 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:06:42.977660 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:06:42.979608 (+ 1948us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:06:42.981025 (+ 1417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:06:42.981032 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:06:42.981033 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:06:42.981256 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:06:42.981458 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:06:42.981459 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:06:42.981459 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:06:42.981551 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:06:42.981753 (+ 202us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:06:42.981757 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:06:42.981865 (+ 108us) client_negotiation.cc:769] Sending connection context
0810 04:06:42.981962 (+ 97us) client_negotiation.cc:240] Negotiation successful
0810 04:06:42.982077 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":107,"threads_started":1}
I20260810 04:06:42.982759 1698 heartbeater.cc:347] Connected to a master server at 127.31.32.62:34751
I20260810 04:06:42.982856 1698 heartbeater.cc:464] Registering TS with master...
I20260810 04:06:42.983047 1698 heartbeater.cc:511] Master 127.31.32.62:34751 requested a full tablet report, sending...
I20260810 04:06:42.983462 1326 ts_manager.cc:194] Registered new tserver with Master: 554b70db09f344a2b8056e8d8bd6bdad (127.31.32.2:35599)
I20260810 04:06:42.983990 1326 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.32.2@KRBTEST.COM'} at 127.31.32.2:38781
I20260810 04:06:42.985344 31872 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskws89Io/build/release/bin/kudu as pid 1558
I20260810 04:06:42.985440 31872 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786334794676589-31872-0/minicluster-data/ts-1/wal/instance
I20260810 04:06:42.985625 31872 external_mini_cluster.cc:1428] Setting key d6b3553d4f1c2fb1ae151f47a0b8be3f
I20260810 04:06:42.989964 1382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:42.983897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.2:39923 (local address 127.31.32.62:34751)
0810 04:06:42.983933 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0810 04:06:42.983935 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:06:42.984094 (+ 159us) server_negotiation.cc:407] Connection header received
0810 04:06:42.984214 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:06:42.984215 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:06:42.984226 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:06:42.984287 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:06:42.984653 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:42.985104 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:42.986113 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:42.986385 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:42.986435 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:06:42.987392 (+ 957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:06:42.987393 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:06:42.987394 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:06:42.987399 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:06:42.988725 (+ 1326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:06:42.989187 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:06:42.989188 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:06:42.989188 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:06:42.989238 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:06:42.989452 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:06:42.989453 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:06:42.989453 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:06:42.989594 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:06:42.989644 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0810 04:06:42.989807 (+ 163us) server_negotiation.cc:299] Negotiation successful
0810 04:06:42.989864 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260810 04:06:42.990046 1704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:42.983802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:34751 (local address 127.31.32.2:39923)
0810 04:06:42.983992 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0810 04:06:42.984003 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 04:06:42.984154 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:06:42.984292 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:06:42.984293 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:06:42.984515 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:06:42.984578 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:06:42.984580 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:06:42.985223 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:06:42.985224 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:06:42.986022 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:06:42.986025 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:06:42.986461 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:06:42.986462 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:06:42.986545 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:06:42.986903 (+ 358us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:06:42.986908 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:06:42.987285 (+ 377us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:06:42.988838 (+ 1553us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:06:42.988839 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:06:42.988839 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:06:42.989097 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:06:42.989302 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:06:42.989303 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:06:42.989303 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:06:42.989373 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:06:42.989658 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:06:42.989659 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:06:42.989707 (+ 48us) client_negotiation.cc:769] Sending connection context
0810 04:06:42.989779 (+ 72us) client_negotiation.cc:240] Negotiation successful
0810 04:06:42.989897 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":107,"threads_started":1}
WARNING: no policy specified for kudu/127.31.32.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.32.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.32.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.3.keytab.
Entry for principal kudu/127.31.32.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.3.keytab.
I20260810 04:06:43.013412 31872 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskws89Io/build/release/bin/kudu
/tmp/dist-test-taskws89Io/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786334794676589-31872-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786334794676589-31872-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786334794676589-31872-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786334794676589-31872-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.32.3:0
--local_ip_for_outbound_sockets=127.31.32.3
--webserver_interface=127.31.32.3
--webserver_port=0
--tserver_master_addrs=127.31.32.62:34751
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.3.keytab
--principal=kudu/127.31.32.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.32.20:41897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786334794676589-31872-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786334794676589-31872-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786334794676589-31872-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:06:43.095990 1707 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:06:43.096251 1707 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:06:43.096277 1707 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:06:43.097348 1707 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:06:43.097381 1707 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:06:43.097401 1707 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260810 04:06:43.097412 1707 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 04:06:43.097429 1707 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.32.3
I20260810 04:06:43.098869 1707 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.32.20:41897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786334794676589-31872-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786334794676589-31872-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.3.keytab
--principal=kudu/127.31.32.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.32.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786334794676589-31872-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.32.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.31.32.62:34751
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1707
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.32.3
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786334794676589-31872-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:06:43.099061 1707 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:06:43.099217 1707 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:06:43.104177 1713 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:06:43.104195 1715 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:06:43.104322 1707 server_base.cc:1034] running on GCE node
W20260810 04:06:43.104223 1717 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:06:43.104586 1707 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:06:43.104842 1707 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:06:43.106011 1707 hybrid_clock.cc:648] HybridClock initialized: now 1786334803105996 us; error 30 us; skew 500 ppm
Aug 10 04:06:43 dist-test-slave-8w9v krb5kdc[1277](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334803, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:06:43.112986 1707 init.cc:377] Logged in from keytab as kudu/127.31.32.3@KRBTEST.COM (short username kudu)
I20260810 04:06:43.113456 1707 webserver.cc:460] Webserver started at http://127.31.32.3:34869/ using document root <none> and password file <none>
I20260810 04:06:43.113703 1707 fs_manager.cc:359] Metadata directory not provided
I20260810 04:06:43.113770 1707 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:06:43.113932 1707 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:06:43.114953 1707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786334794676589-31872-0/minicluster-data/ts-2/data/instance:
uuid: "f935a73e27894272897d653e030f40b1"
format_stamp: "Formatted at 2026-08-10 04:06:43 on dist-test-slave-8w9v"
server_key: "7839a06c011d4fc484435aa044e378c9"
server_key_iv: "7517faaddd68d062db307fdb9b1a72f2"
server_key_version: "encryptionkey@0"
I20260810 04:06:43.115326 1707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786334794676589-31872-0/minicluster-data/ts-2/wal/instance:
uuid: "f935a73e27894272897d653e030f40b1"
format_stamp: "Formatted at 2026-08-10 04:06:43 on dist-test-slave-8w9v"
server_key: "7839a06c011d4fc484435aa044e378c9"
server_key_iv: "7517faaddd68d062db307fdb9b1a72f2"
server_key_version: "encryptionkey@0"
I20260810 04:06:43.116740 1707 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 04:06:43.117437 1727 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:06:43.117599 1707 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 04:06:43.117697 1707 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786334794676589-31872-0/minicluster-data/ts-2/data,/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786334794676589-31872-0/minicluster-data/ts-2/wal
uuid: "f935a73e27894272897d653e030f40b1"
format_stamp: "Formatted at 2026-08-10 04:06:43 on dist-test-slave-8w9v"
server_key: "7839a06c011d4fc484435aa044e378c9"
server_key_iv: "7517faaddd68d062db307fdb9b1a72f2"
server_key_version: "encryptionkey@0"
I20260810 04:06:43.117774 1707 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786334794676589-31872-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786334794676589-31872-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786334794676589-31872-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:06:43.129748 1707 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:06:43.131438 1707 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:06:43.131589 1707 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:06:43.132089 1707 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:06:43.132225 1707 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:06:43.132278 1707 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:06:43.132297 1707 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 04:06:43 dist-test-slave-8w9v krb5kdc[1277](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334803, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.3@KRBTEST.COM for kudu/127.31.32.62@KRBTEST.COM
I20260810 04:06:43.139482 1707 rpc_server.cc:307] RPC server started. Bound to: 127.31.32.3:43747
I20260810 04:06:43.139545 1852 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.32.3:43747 every 8 connection(s)
I20260810 04:06:43.139832 1707 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786334794676589-31872-0/minicluster-data/ts-2/data/info.pb
I20260810 04:06:43.142010 1382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:43.132497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.3:43733 (local address 127.31.32.62:34751)
0810 04:06:43.132541 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0810 04:06:43.132543 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:06:43.132843 (+ 300us) server_negotiation.cc:407] Connection header received
0810 04:06:43.133606 (+ 763us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:06:43.133607 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:06:43.133620 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:06:43.133689 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:06:43.134347 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:43.135012 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:43.136696 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:43.137004 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:43.137086 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:06:43.139383 (+ 2297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:06:43.139385 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:06:43.139385 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:06:43.139393 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:06:43.140970 (+ 1577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:06:43.141290 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:06:43.141291 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:06:43.141291 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:06:43.141340 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:06:43.141531 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:06:43.141532 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:06:43.141532 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:06:43.141641 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:06:43.141690 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0810 04:06:43.141881 (+ 191us) server_negotiation.cc:299] Negotiation successful
0810 04:06:43.141949 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260810 04:06:43.142145 1739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:43.132519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:34751 (local address 127.31.32.3:43733)
0810 04:06:43.132694 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0810 04:06:43.132738 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 04:06:43.133451 (+ 713us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:06:43.133786 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:06:43.133789 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:06:43.134014 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:06:43.134233 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:06:43.134244 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:06:43.135107 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:06:43.135108 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:06:43.136604 (+ 1496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:06:43.136608 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:06:43.137109 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:06:43.137110 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:06:43.137232 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:06:43.137731 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:06:43.137739 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:06:43.139221 (+ 1482us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:06:43.141036 (+ 1815us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:06:43.141040 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:06:43.141041 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:06:43.141229 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:06:43.141394 (+ 165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:06:43.141395 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:06:43.141395 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:06:43.141481 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:06:43.141693 (+ 212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:06:43.141697 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:06:43.141795 (+ 98us) client_negotiation.cc:769] Sending connection context
0810 04:06:43.141871 (+ 76us) client_negotiation.cc:240] Negotiation successful
0810 04:06:43.141978 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":82,"threads_started":1}
I20260810 04:06:43.142649 1853 heartbeater.cc:347] Connected to a master server at 127.31.32.62:34751
I20260810 04:06:43.142722 1853 heartbeater.cc:464] Registering TS with master...
I20260810 04:06:43.142922 1853 heartbeater.cc:511] Master 127.31.32.62:34751 requested a full tablet report, sending...
I20260810 04:06:43.143236 1325 ts_manager.cc:194] Registered new tserver with Master: f935a73e27894272897d653e030f40b1 (127.31.32.3:43747)
I20260810 04:06:43.143592 1325 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.32.3@KRBTEST.COM'} at 127.31.32.3:43733
I20260810 04:06:43.148034 1382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:43.143658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.3:34011 (local address 127.31.32.62:34751)
0810 04:06:43.143696 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0810 04:06:43.143698 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:06:43.143812 (+ 114us) server_negotiation.cc:407] Connection header received
0810 04:06:43.143865 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:06:43.143866 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:06:43.143875 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:06:43.143922 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:06:43.144289 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:43.144698 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:43.145297 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:43.145468 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:43.145518 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:06:43.146220 (+ 702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:06:43.146221 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:06:43.146221 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:06:43.146227 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:06:43.147260 (+ 1033us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:06:43.147556 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:06:43.147556 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:06:43.147556 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:06:43.147601 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:06:43.147711 (+ 110us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:06:43.147712 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:06:43.147712 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:06:43.147787 (+ 75us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:06:43.147850 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0810 04:06:43.147939 (+ 89us) server_negotiation.cc:299] Negotiation successful
0810 04:06:43.147987 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260810 04:06:43.148038 1860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:43.143598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:34751 (local address 127.31.32.3:34011)
0810 04:06:43.143739 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0810 04:06:43.143748 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 04:06:43.143821 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:06:43.143941 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:06:43.143942 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:06:43.144173 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:06:43.144226 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:06:43.144228 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:06:43.144780 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:06:43.144781 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:06:43.145240 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:06:43.145242 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:06:43.145533 (+ 291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:06:43.145534 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:06:43.145592 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:06:43.145897 (+ 305us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:06:43.145901 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:06:43.146149 (+ 248us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:06:43.147324 (+ 1175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:06:43.147325 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:06:43.147325 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:06:43.147494 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:06:43.147640 (+ 146us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:06:43.147640 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:06:43.147640 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:06:43.147681 (+ 41us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:06:43.147852 (+ 171us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:06:43.147852 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:06:43.147882 (+ 30us) client_negotiation.cc:769] Sending connection context
0810 04:06:43.147923 (+ 41us) client_negotiation.cc:240] Negotiation successful
0810 04:06:43.147985 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":73,"threads_started":1}
I20260810 04:06:43.148808 31872 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskws89Io/build/release/bin/kudu as pid 1707
I20260810 04:06:43.148886 31872 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786334794676589-31872-0/minicluster-data/ts-2/wal/instance
I20260810 04:06:43.149116 31872 external_mini_cluster.cc:1428] Setting key 52138a462b3765eeae69708a6ec952e3
I20260810 04:06:43.150040 31872 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 10 04:06:43 dist-test-slave-8w9v krb5kdc[1277](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334803, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 10 04:06:43 dist-test-slave-8w9v krb5kdc[1277](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334803, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.32.62@KRBTEST.COM
I20260810 04:06:43.172669 1382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:43.166276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44238 (local address 127.31.32.62:34751)
0810 04:06:43.166331 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0810 04:06:43.166332 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 04:06:43.166355 (+ 23us) server_negotiation.cc:407] Connection header received
0810 04:06:43.166413 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:06:43.166414 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:06:43.166421 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:06:43.166470 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:06:43.166983 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:43.167471 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:43.168241 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:43.168429 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:43.168483 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:06:43.170573 (+ 2090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:06:43.170574 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:06:43.170574 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:06:43.170580 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:06:43.171711 (+ 1131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:06:43.172027 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:06:43.172028 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:06:43.172028 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:06:43.172074 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:06:43.172281 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:06:43.172282 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:06:43.172282 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:06:43.172389 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:06:43.172460 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0810 04:06:43.172558 (+ 98us) server_negotiation.cc:299] Negotiation successful
0810 04:06:43.172611 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260810 04:06:43.174190 1325 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44238:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 04:06:43.175782 1325 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 04:06:43.182516 1871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:43.178075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.3:43747 (local address 127.0.0.1:57182)
0810 04:06:43.178416 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0810 04:06:43.178427 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 04:06:43.178478 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:06:43.178857 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:06:43.178858 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:06:43.178860 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:06:43.178903 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:06:43.178904 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:06:43.180254 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:06:43.180255 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:06:43.181356 (+ 1101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:06:43.181359 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:06:43.182213 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:06:43.182214 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:06:43.182311 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:06:43.182324 (+ 13us) client_negotiation.cc:769] Sending connection context
0810 04:06:43.182400 (+ 76us) client_negotiation.cc:240] Negotiation successful
0810 04:06:43.182444 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"mutex_wait_us":27,"thread_start_us":60,"threads_started":1}
I20260810 04:06:43.182794 1875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:43.178377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57182 (local address 127.31.32.3:43747)
0810 04:06:43.178600 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0810 04:06:43.178605 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 04:06:43.178631 (+ 26us) server_negotiation.cc:407] Connection header received
0810 04:06:43.178667 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:06:43.178669 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:06:43.178683 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:06:43.179007 (+ 324us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:06:43.179094 (+ 87us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:43.180196 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:43.181443 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:43.182122 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:43.182198 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:06:43.182253 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0810 04:06:43.182427 (+ 174us) server_negotiation.cc:299] Negotiation successful
0810 04:06:43.182477 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":80,"threads_started":1}
I20260810 04:06:43.183022 1873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:43.178160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44436 (local address 127.31.32.1:33943)
0810 04:06:43.178353 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0810 04:06:43.178358 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 04:06:43.178368 (+ 10us) server_negotiation.cc:407] Connection header received
0810 04:06:43.178468 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:06:43.178470 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:06:43.178604 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:06:43.178703 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:06:43.178884 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:43.180073 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:43.181675 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:43.182424 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:43.182489 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:06:43.182547 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0810 04:06:43.182887 (+ 340us) server_negotiation.cc:299] Negotiation successful
0810 04:06:43.182929 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":86,"threads_started":1}
I20260810 04:06:43.183172 1872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:43.178074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.1:33943 (local address 127.0.0.1:44436)
0810 04:06:43.178216 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0810 04:06:43.178260 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 04:06:43.178327 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:06:43.178725 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:06:43.178728 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:06:43.178730 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:06:43.178798 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:06:43.178800 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:06:43.180205 (+ 1405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:06:43.180206 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:06:43.181592 (+ 1386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:06:43.181594 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:06:43.182596 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:06:43.182597 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:06:43.182687 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:06:43.182690 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 04:06:43.182746 (+ 56us) client_negotiation.cc:240] Negotiation successful
0810 04:06:43.182787 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"mutex_wait_us":39,"thread_start_us":82,"threads_started":1}
I20260810 04:06:43.183463 1782 tablet_service.cc:1467] Processing CreateTablet for tablet 7b18c69319c44be59588c14519944b4d (DEFAULT_TABLE table=test-table [id=215df5b718154f86aca79b4c9874801a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:06:43.183887 1782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b18c69319c44be59588c14519944b4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:06:43.183795 1476 tablet_service.cc:1467] Processing CreateTablet for tablet 7b18c69319c44be59588c14519944b4d (DEFAULT_TABLE table=test-table [id=215df5b718154f86aca79b4c9874801a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:06:43.183990 1874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:43.178275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.2:35599 (local address 127.0.0.1:41030)
0810 04:06:43.180302 (+ 2027us) negotiation.cc:107] Waiting for socket to connect
0810 04:06:43.180311 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 04:06:43.180361 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:06:43.180540 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:06:43.180541 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:06:43.180543 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:06:43.180580 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:06:43.180581 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:06:43.181732 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:06:43.181732 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:06:43.182662 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:06:43.182665 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:06:43.183721 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:06:43.183722 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:06:43.183785 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:06:43.183789 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 04:06:43.183850 (+ 61us) client_negotiation.cc:240] Negotiation successful
0810 04:06:43.183886 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2003,"thread_start_us":92,"threads_started":1}
I20260810 04:06:43.184015 1876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:43.178655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41030 (local address 127.31.32.2:35599)
0810 04:06:43.178963 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0810 04:06:43.178968 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 04:06:43.180377 (+ 1409us) server_negotiation.cc:407] Connection header received
0810 04:06:43.180427 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:06:43.180429 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:06:43.180445 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:06:43.180529 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:06:43.180640 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:43.181663 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:43.182798 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:43.183618 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:43.183709 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:06:43.183762 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0810 04:06:43.183881 (+ 119us) server_negotiation.cc:299] Negotiation successful
0810 04:06:43.183923 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":114,"threads_started":1}
I20260810 04:06:43.184183 1476 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b18c69319c44be59588c14519944b4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:06:43.184584 1629 tablet_service.cc:1467] Processing CreateTablet for tablet 7b18c69319c44be59588c14519944b4d (DEFAULT_TABLE table=test-table [id=215df5b718154f86aca79b4c9874801a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:06:43.184907 1629 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b18c69319c44be59588c14519944b4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:06:43.187042 1878 tablet_bootstrap.cc:492] T 7b18c69319c44be59588c14519944b4d P 3fd00917592c4e1e91b1e32810acd65a: Bootstrap starting.
I20260810 04:06:43.187700 1878 tablet_bootstrap.cc:654] T 7b18c69319c44be59588c14519944b4d P 3fd00917592c4e1e91b1e32810acd65a: Neither blocks nor log segments found. Creating new log.
I20260810 04:06:43.188058 1878 log.cc:824] T 7b18c69319c44be59588c14519944b4d P 3fd00917592c4e1e91b1e32810acd65a: Log is configured to *not* fsync() on all Append() calls
I20260810 04:06:43.188324 1880 tablet_bootstrap.cc:492] T 7b18c69319c44be59588c14519944b4d P 554b70db09f344a2b8056e8d8bd6bdad: Bootstrap starting.
I20260810 04:06:43.188489 1879 tablet_bootstrap.cc:492] T 7b18c69319c44be59588c14519944b4d P f935a73e27894272897d653e030f40b1: Bootstrap starting.
I20260810 04:06:43.188778 1878 tablet_bootstrap.cc:492] T 7b18c69319c44be59588c14519944b4d P 3fd00917592c4e1e91b1e32810acd65a: No bootstrap required, opened a new log
I20260810 04:06:43.188863 1878 ts_tablet_manager.cc:1397] T 7b18c69319c44be59588c14519944b4d P 3fd00917592c4e1e91b1e32810acd65a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 04:06:43.189220 1880 tablet_bootstrap.cc:654] T 7b18c69319c44be59588c14519944b4d P 554b70db09f344a2b8056e8d8bd6bdad: Neither blocks nor log segments found. Creating new log.
I20260810 04:06:43.189452 1879 tablet_bootstrap.cc:654] T 7b18c69319c44be59588c14519944b4d P f935a73e27894272897d653e030f40b1: Neither blocks nor log segments found. Creating new log.
I20260810 04:06:43.189714 1880 log.cc:824] T 7b18c69319c44be59588c14519944b4d P 554b70db09f344a2b8056e8d8bd6bdad: Log is configured to *not* fsync() on all Append() calls
I20260810 04:06:43.189791 1879 log.cc:824] T 7b18c69319c44be59588c14519944b4d P f935a73e27894272897d653e030f40b1: Log is configured to *not* fsync() on all Append() calls
I20260810 04:06:43.190562 1879 tablet_bootstrap.cc:492] T 7b18c69319c44be59588c14519944b4d P f935a73e27894272897d653e030f40b1: No bootstrap required, opened a new log
I20260810 04:06:43.190634 1879 ts_tablet_manager.cc:1397] T 7b18c69319c44be59588c14519944b4d P f935a73e27894272897d653e030f40b1: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 04:06:43.190721 1880 tablet_bootstrap.cc:492] T 7b18c69319c44be59588c14519944b4d P 554b70db09f344a2b8056e8d8bd6bdad: No bootstrap required, opened a new log
I20260810 04:06:43.190793 1880 ts_tablet_manager.cc:1397] T 7b18c69319c44be59588c14519944b4d P 554b70db09f344a2b8056e8d8bd6bdad: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260810 04:06:43.190749 1878 raft_consensus.cc:348] T 7b18c69319c44be59588c14519944b4d P 3fd00917592c4e1e91b1e32810acd65a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fd00917592c4e1e91b1e32810acd65a" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 33943 } } peers { permanent_uuid: "554b70db09f344a2b8056e8d8bd6bdad" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 35599 } } peers { permanent_uuid: "f935a73e27894272897d653e030f40b1" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 43747 } }
I20260810 04:06:43.190932 1878 raft_consensus.cc:374] T 7b18c69319c44be59588c14519944b4d P 3fd00917592c4e1e91b1e32810acd65a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:06:43.190974 1878 raft_consensus.cc:729] T 7b18c69319c44be59588c14519944b4d P 3fd00917592c4e1e91b1e32810acd65a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fd00917592c4e1e91b1e32810acd65a, State: Initialized, Role: FOLLOWER
I20260810 04:06:43.191082 1878 consensus_queue.cc:260] T 7b18c69319c44be59588c14519944b4d P 3fd00917592c4e1e91b1e32810acd65a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fd00917592c4e1e91b1e32810acd65a" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 33943 } } peers { permanent_uuid: "554b70db09f344a2b8056e8d8bd6bdad" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 35599 } } peers { permanent_uuid: "f935a73e27894272897d653e030f40b1" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 43747 } }
I20260810 04:06:43.191335 1878 ts_tablet_manager.cc:1428] T 7b18c69319c44be59588c14519944b4d P 3fd00917592c4e1e91b1e32810acd65a: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260810 04:06:43.191386 1549 heartbeater.cc:503] Master 127.31.32.62:34751 was elected leader, sending a full tablet report...
I20260810 04:06:43.192281 1879 raft_consensus.cc:348] T 7b18c69319c44be59588c14519944b4d P f935a73e27894272897d653e030f40b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fd00917592c4e1e91b1e32810acd65a" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 33943 } } peers { permanent_uuid: "554b70db09f344a2b8056e8d8bd6bdad" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 35599 } } peers { permanent_uuid: "f935a73e27894272897d653e030f40b1" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 43747 } }
I20260810 04:06:43.192399 1879 raft_consensus.cc:374] T 7b18c69319c44be59588c14519944b4d P f935a73e27894272897d653e030f40b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:06:43.192420 1879 raft_consensus.cc:729] T 7b18c69319c44be59588c14519944b4d P f935a73e27894272897d653e030f40b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f935a73e27894272897d653e030f40b1, State: Initialized, Role: FOLLOWER
I20260810 04:06:43.192516 1879 consensus_queue.cc:260] T 7b18c69319c44be59588c14519944b4d P f935a73e27894272897d653e030f40b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fd00917592c4e1e91b1e32810acd65a" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 33943 } } peers { permanent_uuid: "554b70db09f344a2b8056e8d8bd6bdad" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 35599 } } peers { permanent_uuid: "f935a73e27894272897d653e030f40b1" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 43747 } }
I20260810 04:06:43.192764 1879 ts_tablet_manager.cc:1428] T 7b18c69319c44be59588c14519944b4d P f935a73e27894272897d653e030f40b1: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260810 04:06:43.192790 1853 heartbeater.cc:503] Master 127.31.32.62:34751 was elected leader, sending a full tablet report...
I20260810 04:06:43.192862 1880 raft_consensus.cc:348] T 7b18c69319c44be59588c14519944b4d P 554b70db09f344a2b8056e8d8bd6bdad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fd00917592c4e1e91b1e32810acd65a" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 33943 } } peers { permanent_uuid: "554b70db09f344a2b8056e8d8bd6bdad" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 35599 } } peers { permanent_uuid: "f935a73e27894272897d653e030f40b1" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 43747 } }
I20260810 04:06:43.193055 1880 raft_consensus.cc:374] T 7b18c69319c44be59588c14519944b4d P 554b70db09f344a2b8056e8d8bd6bdad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:06:43.193101 1880 raft_consensus.cc:729] T 7b18c69319c44be59588c14519944b4d P 554b70db09f344a2b8056e8d8bd6bdad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 554b70db09f344a2b8056e8d8bd6bdad, State: Initialized, Role: FOLLOWER
I20260810 04:06:43.193204 1880 consensus_queue.cc:260] T 7b18c69319c44be59588c14519944b4d P 554b70db09f344a2b8056e8d8bd6bdad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fd00917592c4e1e91b1e32810acd65a" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 33943 } } peers { permanent_uuid: "554b70db09f344a2b8056e8d8bd6bdad" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 35599 } } peers { permanent_uuid: "f935a73e27894272897d653e030f40b1" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 43747 } }
I20260810 04:06:43.193498 1880 ts_tablet_manager.cc:1428] T 7b18c69319c44be59588c14519944b4d P 554b70db09f344a2b8056e8d8bd6bdad: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260810 04:06:43.193552 1698 heartbeater.cc:503] Master 127.31.32.62:34751 was elected leader, sending a full tablet report...
W20260810 04:06:43.229843 1699 tablet.cc:2369] T 7b18c69319c44be59588c14519944b4d P 554b70db09f344a2b8056e8d8bd6bdad: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 04:06:43.315341 1886 raft_consensus.cc:482] T 7b18c69319c44be59588c14519944b4d P f935a73e27894272897d653e030f40b1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 04:06:43.315493 1886 raft_consensus.cc:504] T 7b18c69319c44be59588c14519944b4d P f935a73e27894272897d653e030f40b1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fd00917592c4e1e91b1e32810acd65a" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 33943 } } peers { permanent_uuid: "554b70db09f344a2b8056e8d8bd6bdad" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 35599 } } peers { permanent_uuid: "f935a73e27894272897d653e030f40b1" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 43747 } }
I20260810 04:06:43.315866 1886 leader_election.cc:288] T 7b18c69319c44be59588c14519944b4d P f935a73e27894272897d653e030f40b1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fd00917592c4e1e91b1e32810acd65a (127.31.32.1:33943), 554b70db09f344a2b8056e8d8bd6bdad (127.31.32.2:35599)
I20260810 04:06:43.318928 1873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:43.316213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.3:54111 (local address 127.31.32.1:33943)
0810 04:06:43.316261 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0810 04:06:43.316263 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:06:43.316272 (+ 9us) server_negotiation.cc:407] Connection header received
0810 04:06:43.316324 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:06:43.316324 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:06:43.316335 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:06:43.316414 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:06:43.316557 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:43.317315 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:43.318192 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:43.318577 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:43.318660 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:06:43.318670 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0810 04:06:43.318817 (+ 147us) server_negotiation.cc:299] Negotiation successful
0810 04:06:43.318859 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260810 04:06:43.318928 1739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:43.316097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.1:33943 (local address 127.31.32.3:54111)
0810 04:06:43.316181 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0810 04:06:43.316195 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 04:06:43.316255 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:06:43.316426 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:06:43.316427 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:06:43.316430 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:06:43.316500 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:06:43.316502 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:06:43.317392 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:06:43.317393 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:06:43.318101 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:06:43.318103 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:06:43.318658 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:06:43.318659 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:06:43.318741 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:06:43.318744 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 04:06:43.318808 (+ 64us) client_negotiation.cc:240] Negotiation successful
0810 04:06:43.318858 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260810 04:06:43.319253 1876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:43.316341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.3:42115 (local address 127.31.32.2:35599)
0810 04:06:43.316410 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0810 04:06:43.316412 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:06:43.316692 (+ 280us) server_negotiation.cc:407] Connection header received
0810 04:06:43.316720 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:06:43.316721 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:06:43.316734 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:06:43.316790 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:06:43.316938 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:43.317548 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:43.318502 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:43.318873 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:43.318950 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:06:43.318960 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0810 04:06:43.319154 (+ 194us) server_negotiation.cc:299] Negotiation successful
0810 04:06:43.319188 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260810 04:06:43.319273 1484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b18c69319c44be59588c14519944b4d" candidate_uuid: "f935a73e27894272897d653e030f40b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fd00917592c4e1e91b1e32810acd65a" is_pre_election: true
I20260810 04:06:43.319443 1484 raft_consensus.cc:2454] T 7b18c69319c44be59588c14519944b4d P 3fd00917592c4e1e91b1e32810acd65a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f935a73e27894272897d653e030f40b1 in term 0.
I20260810 04:06:43.319509 1888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:43.316256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.2:35599 (local address 127.31.32.3:42115)
0810 04:06:43.316592 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0810 04:06:43.316600 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0810 04:06:43.316645 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:06:43.316802 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:06:43.316803 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:06:43.316804 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:06:43.316874 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:06:43.316875 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:06:43.317626 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:06:43.317628 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:06:43.318413 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:06:43.318421 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:06:43.318972 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:06:43.318974 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:06:43.319063 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:06:43.319068 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 04:06:43.319349 (+ 281us) client_negotiation.cc:240] Negotiation successful
0810 04:06:43.319400 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":141,"threads_started":1}
I20260810 04:06:43.319713 1732 leader_election.cc:302] T 7b18c69319c44be59588c14519944b4d P f935a73e27894272897d653e030f40b1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fd00917592c4e1e91b1e32810acd65a, f935a73e27894272897d653e030f40b1; no voters:
I20260810 04:06:43.319820 1650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b18c69319c44be59588c14519944b4d" candidate_uuid: "f935a73e27894272897d653e030f40b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "554b70db09f344a2b8056e8d8bd6bdad" is_pre_election: true
I20260810 04:06:43.319923 1886 raft_consensus.cc:2781] T 7b18c69319c44be59588c14519944b4d P f935a73e27894272897d653e030f40b1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 04:06:43.319983 1650 raft_consensus.cc:2454] T 7b18c69319c44be59588c14519944b4d P 554b70db09f344a2b8056e8d8bd6bdad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f935a73e27894272897d653e030f40b1 in term 0.
I20260810 04:06:43.319995 1886 raft_consensus.cc:482] T 7b18c69319c44be59588c14519944b4d P f935a73e27894272897d653e030f40b1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 04:06:43.320029 1886 raft_consensus.cc:3037] T 7b18c69319c44be59588c14519944b4d P f935a73e27894272897d653e030f40b1 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:06:43.320906 1886 raft_consensus.cc:504] T 7b18c69319c44be59588c14519944b4d P f935a73e27894272897d653e030f40b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fd00917592c4e1e91b1e32810acd65a" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 33943 } } peers { permanent_uuid: "554b70db09f344a2b8056e8d8bd6bdad" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 35599 } } peers { permanent_uuid: "f935a73e27894272897d653e030f40b1" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 43747 } }
I20260810 04:06:43.321063 1886 leader_election.cc:288] T 7b18c69319c44be59588c14519944b4d P f935a73e27894272897d653e030f40b1 [CANDIDATE]: Term 1 election: Requested vote from peers 3fd00917592c4e1e91b1e32810acd65a (127.31.32.1:33943), 554b70db09f344a2b8056e8d8bd6bdad (127.31.32.2:35599)
I20260810 04:06:43.321203 1484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b18c69319c44be59588c14519944b4d" candidate_uuid: "f935a73e27894272897d653e030f40b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fd00917592c4e1e91b1e32810acd65a"
I20260810 04:06:43.321240 1650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b18c69319c44be59588c14519944b4d" candidate_uuid: "f935a73e27894272897d653e030f40b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "554b70db09f344a2b8056e8d8bd6bdad"
I20260810 04:06:43.321300 1484 raft_consensus.cc:3037] T 7b18c69319c44be59588c14519944b4d P 3fd00917592c4e1e91b1e32810acd65a [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:06:43.321307 1650 raft_consensus.cc:3037] T 7b18c69319c44be59588c14519944b4d P 554b70db09f344a2b8056e8d8bd6bdad [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:06:43.322162 1484 raft_consensus.cc:2454] T 7b18c69319c44be59588c14519944b4d P 3fd00917592c4e1e91b1e32810acd65a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f935a73e27894272897d653e030f40b1 in term 1.
I20260810 04:06:43.322162 1650 raft_consensus.cc:2454] T 7b18c69319c44be59588c14519944b4d P 554b70db09f344a2b8056e8d8bd6bdad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f935a73e27894272897d653e030f40b1 in term 1.
I20260810 04:06:43.322327 1732 leader_election.cc:302] T 7b18c69319c44be59588c14519944b4d P f935a73e27894272897d653e030f40b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fd00917592c4e1e91b1e32810acd65a, f935a73e27894272897d653e030f40b1; no voters:
I20260810 04:06:43.322433 1886 raft_consensus.cc:2781] T 7b18c69319c44be59588c14519944b4d P f935a73e27894272897d653e030f40b1 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:06:43.322580 1886 raft_consensus.cc:686] T 7b18c69319c44be59588c14519944b4d P f935a73e27894272897d653e030f40b1 [term 1 LEADER]: Becoming Leader. State: Replica: f935a73e27894272897d653e030f40b1, State: Running, Role: LEADER
I20260810 04:06:43.322674 1886 consensus_queue.cc:237] T 7b18c69319c44be59588c14519944b4d P f935a73e27894272897d653e030f40b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fd00917592c4e1e91b1e32810acd65a" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 33943 } } peers { permanent_uuid: "554b70db09f344a2b8056e8d8bd6bdad" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 35599 } } peers { permanent_uuid: "f935a73e27894272897d653e030f40b1" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 43747 } }
I20260810 04:06:43.323483 1325 catalog_manager.cc:5495] T 7b18c69319c44be59588c14519944b4d P f935a73e27894272897d653e030f40b1 reported cstate change: term changed from 0 to 1, leader changed from <none> to f935a73e27894272897d653e030f40b1 (127.31.32.3). New cstate: current_term: 1 leader_uuid: "f935a73e27894272897d653e030f40b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fd00917592c4e1e91b1e32810acd65a" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 33943 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "554b70db09f344a2b8056e8d8bd6bdad" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 35599 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f935a73e27894272897d653e030f40b1" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 43747 } health_report { overall_health: HEALTHY } } }
W20260810 04:06:43.331105 1550 tablet.cc:2369] T 7b18c69319c44be59588c14519944b4d P 3fd00917592c4e1e91b1e32810acd65a: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 10 04:06:43 dist-test-slave-8w9v krb5kdc[1277](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334802, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.62@KRBTEST.COM for kudu/127.31.32.62@KRBTEST.COM
I20260810 04:06:43.347661 1382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:43.340755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44248 (local address 127.31.32.62:34751)
0810 04:06:43.340824 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0810 04:06:43.340826 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:06:43.340890 (+ 64us) server_negotiation.cc:407] Connection header received
0810 04:06:43.340953 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:06:43.340954 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:06:43.340964 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:06:43.341022 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:06:43.341419 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:43.341873 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:43.342613 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:43.342791 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:43.342865 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:06:43.345036 (+ 2171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:06:43.345037 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:06:43.345038 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:06:43.345044 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:06:43.346279 (+ 1235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:06:43.346677 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:06:43.346678 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:06:43.346679 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:06:43.346735 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:06:43.347038 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:06:43.347040 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:06:43.347040 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:06:43.347170 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:06:43.347254 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0810 04:06:43.347421 (+ 167us) server_negotiation.cc:299] Negotiation successful
0810 04:06:43.347532 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260810 04:06:43.347752 1896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:43.340670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:34751 (local address 127.0.0.1:44248)
0810 04:06:43.340819 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0810 04:06:43.340829 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 04:06:43.340891 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:06:43.341033 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:06:43.341033 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:06:43.341303 (+ 270us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:06:43.341349 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:06:43.341351 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:06:43.341939 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:06:43.341939 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:06:43.342514 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:06:43.342516 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:06:43.342865 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:06:43.342865 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:06:43.342921 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:06:43.343457 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:06:43.343462 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:06:43.344948 (+ 1486us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:06:43.346398 (+ 1450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:06:43.346406 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:06:43.346408 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:06:43.346597 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:06:43.346805 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:06:43.346806 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:06:43.346806 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:06:43.346875 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:06:43.347244 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:06:43.347252 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:06:43.347301 (+ 49us) client_negotiation.cc:769] Sending connection context
0810 04:06:43.347397 (+ 96us) client_negotiation.cc:240] Negotiation successful
0810 04:06:43.347596 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"mutex_wait_us":96,"thread_start_us":67,"threads_started":1}
I20260810 04:06:43.349617 1325 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.31.32.62@KRBTEST.COM'} at 127.0.0.1:44248:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260810 04:06:43.349815 1325 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 04:06:43.351161 1476 tablet_service.cc:1467] Processing CreateTablet for tablet dfc4af3152e446fcb8c1259adff6fc21 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e1914b5a0ca346b9941f79a10f6b1f20]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 04:06:43.351284 1629 tablet_service.cc:1467] Processing CreateTablet for tablet dfc4af3152e446fcb8c1259adff6fc21 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e1914b5a0ca346b9941f79a10f6b1f20]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 04:06:43.351369 1476 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfc4af3152e446fcb8c1259adff6fc21. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:06:43.351419 1629 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfc4af3152e446fcb8c1259adff6fc21. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:06:43.351425 1782 tablet_service.cc:1467] Processing CreateTablet for tablet dfc4af3152e446fcb8c1259adff6fc21 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e1914b5a0ca346b9941f79a10f6b1f20]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 04:06:43.351595 1782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfc4af3152e446fcb8c1259adff6fc21. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:06:43.352917 1878 tablet_bootstrap.cc:492] T dfc4af3152e446fcb8c1259adff6fc21 P 3fd00917592c4e1e91b1e32810acd65a: Bootstrap starting.
I20260810 04:06:43.352986 1879 tablet_bootstrap.cc:492] T dfc4af3152e446fcb8c1259adff6fc21 P f935a73e27894272897d653e030f40b1: Bootstrap starting.
I20260810 04:06:43.353286 1878 tablet_bootstrap.cc:654] T dfc4af3152e446fcb8c1259adff6fc21 P 3fd00917592c4e1e91b1e32810acd65a: Neither blocks nor log segments found. Creating new log.
I20260810 04:06:43.353336 1879 tablet_bootstrap.cc:654] T dfc4af3152e446fcb8c1259adff6fc21 P f935a73e27894272897d653e030f40b1: Neither blocks nor log segments found. Creating new log.
I20260810 04:06:43.353600 1880 tablet_bootstrap.cc:492] T dfc4af3152e446fcb8c1259adff6fc21 P 554b70db09f344a2b8056e8d8bd6bdad: Bootstrap starting.
I20260810 04:06:43.353961 1878 tablet_bootstrap.cc:492] T dfc4af3152e446fcb8c1259adff6fc21 P 3fd00917592c4e1e91b1e32810acd65a: No bootstrap required, opened a new log
I20260810 04:06:43.354005 1878 ts_tablet_manager.cc:1397] T dfc4af3152e446fcb8c1259adff6fc21 P 3fd00917592c4e1e91b1e32810acd65a: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260810 04:06:43.354015 1879 tablet_bootstrap.cc:492] T dfc4af3152e446fcb8c1259adff6fc21 P f935a73e27894272897d653e030f40b1: No bootstrap required, opened a new log
I20260810 04:06:43.354051 1879 ts_tablet_manager.cc:1397] T dfc4af3152e446fcb8c1259adff6fc21 P f935a73e27894272897d653e030f40b1: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260810 04:06:43.354179 1878 raft_consensus.cc:348] T dfc4af3152e446fcb8c1259adff6fc21 P 3fd00917592c4e1e91b1e32810acd65a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fd00917592c4e1e91b1e32810acd65a" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 33943 } } peers { permanent_uuid: "554b70db09f344a2b8056e8d8bd6bdad" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 35599 } } peers { permanent_uuid: "f935a73e27894272897d653e030f40b1" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 43747 } }
I20260810 04:06:43.354195 1879 raft_consensus.cc:348] T dfc4af3152e446fcb8c1259adff6fc21 P f935a73e27894272897d653e030f40b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fd00917592c4e1e91b1e32810acd65a" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 33943 } } peers { permanent_uuid: "554b70db09f344a2b8056e8d8bd6bdad" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 35599 } } peers { permanent_uuid: "f935a73e27894272897d653e030f40b1" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 43747 } }
I20260810 04:06:43.354243 1878 raft_consensus.cc:374] T dfc4af3152e446fcb8c1259adff6fc21 P 3fd00917592c4e1e91b1e32810acd65a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:06:43.354247 1879 raft_consensus.cc:374] T dfc4af3152e446fcb8c1259adff6fc21 P f935a73e27894272897d653e030f40b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:06:43.354267 1879 raft_consensus.cc:729] T dfc4af3152e446fcb8c1259adff6fc21 P f935a73e27894272897d653e030f40b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f935a73e27894272897d653e030f40b1, State: Initialized, Role: FOLLOWER
I20260810 04:06:43.354267 1878 raft_consensus.cc:729] T dfc4af3152e446fcb8c1259adff6fc21 P 3fd00917592c4e1e91b1e32810acd65a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fd00917592c4e1e91b1e32810acd65a, State: Initialized, Role: FOLLOWER
I20260810 04:06:43.354301 1880 tablet_bootstrap.cc:654] T dfc4af3152e446fcb8c1259adff6fc21 P 554b70db09f344a2b8056e8d8bd6bdad: Neither blocks nor log segments found. Creating new log.
I20260810 04:06:43.354327 1878 consensus_queue.cc:260] T dfc4af3152e446fcb8c1259adff6fc21 P 3fd00917592c4e1e91b1e32810acd65a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fd00917592c4e1e91b1e32810acd65a" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 33943 } } peers { permanent_uuid: "554b70db09f344a2b8056e8d8bd6bdad" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 35599 } } peers { permanent_uuid: "f935a73e27894272897d653e030f40b1" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 43747 } }
I20260810 04:06:43.354326 1879 consensus_queue.cc:260] T dfc4af3152e446fcb8c1259adff6fc21 P f935a73e27894272897d653e030f40b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fd00917592c4e1e91b1e32810acd65a" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 33943 } } peers { permanent_uuid: "554b70db09f344a2b8056e8d8bd6bdad" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 35599 } } peers { permanent_uuid: "f935a73e27894272897d653e030f40b1" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 43747 } }
I20260810 04:06:43.354427 1878 ts_tablet_manager.cc:1428] T dfc4af3152e446fcb8c1259adff6fc21 P 3fd00917592c4e1e91b1e32810acd65a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260810 04:06:43.354435 1879 ts_tablet_manager.cc:1428] T dfc4af3152e446fcb8c1259adff6fc21 P f935a73e27894272897d653e030f40b1: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260810 04:06:43.354461 1886 tablet_replica.cc:439] T dfc4af3152e446fcb8c1259adff6fc21 P f935a73e27894272897d653e030f40b1: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fd00917592c4e1e91b1e32810acd65a" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 33943 } } peers { permanent_uuid: "554b70db09f344a2b8056e8d8bd6bdad" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 35599 } } peers { permanent_uuid: "f935a73e27894272897d653e030f40b1" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 43747 } } }
I20260810 04:06:43.354557 1886 tablet_replica.cc:442] T dfc4af3152e446fcb8c1259adff6fc21 P f935a73e27894272897d653e030f40b1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 04:06:43.354681 1885 tablet_replica.cc:439] T dfc4af3152e446fcb8c1259adff6fc21 P 3fd00917592c4e1e91b1e32810acd65a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fd00917592c4e1e91b1e32810acd65a" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 33943 } } peers { permanent_uuid: "554b70db09f344a2b8056e8d8bd6bdad" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 35599 } } peers { permanent_uuid: "f935a73e27894272897d653e030f40b1" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 43747 } } }
I20260810 04:06:43.354773 1885 tablet_replica.cc:442] T dfc4af3152e446fcb8c1259adff6fc21 P 3fd00917592c4e1e91b1e32810acd65a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 04:06:43.355860 1880 tablet_bootstrap.cc:492] T dfc4af3152e446fcb8c1259adff6fc21 P 554b70db09f344a2b8056e8d8bd6bdad: No bootstrap required, opened a new log
I20260810 04:06:43.355906 1880 ts_tablet_manager.cc:1397] T dfc4af3152e446fcb8c1259adff6fc21 P 554b70db09f344a2b8056e8d8bd6bdad: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260810 04:06:43.356081 1880 raft_consensus.cc:348] T dfc4af3152e446fcb8c1259adff6fc21 P 554b70db09f344a2b8056e8d8bd6bdad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fd00917592c4e1e91b1e32810acd65a" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 33943 } } peers { permanent_uuid: "554b70db09f344a2b8056e8d8bd6bdad" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 35599 } } peers { permanent_uuid: "f935a73e27894272897d653e030f40b1" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 43747 } }
I20260810 04:06:43.356200 1880 raft_consensus.cc:374] T dfc4af3152e446fcb8c1259adff6fc21 P 554b70db09f344a2b8056e8d8bd6bdad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:06:43.356232 1880 raft_consensus.cc:729] T dfc4af3152e446fcb8c1259adff6fc21 P 554b70db09f344a2b8056e8d8bd6bdad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 554b70db09f344a2b8056e8d8bd6bdad, State: Initialized, Role: FOLLOWER
I20260810 04:06:43.356309 1880 consensus_queue.cc:260] T dfc4af3152e446fcb8c1259adff6fc21 P 554b70db09f344a2b8056e8d8bd6bdad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fd00917592c4e1e91b1e32810acd65a" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 33943 } } peers { permanent_uuid: "554b70db09f344a2b8056e8d8bd6bdad" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 35599 } } peers { permanent_uuid: "f935a73e27894272897d653e030f40b1" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 43747 } }
I20260810 04:06:43.356423 1880 ts_tablet_manager.cc:1428] T dfc4af3152e446fcb8c1259adff6fc21 P 554b70db09f344a2b8056e8d8bd6bdad: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260810 04:06:43.356484 1887 tablet_replica.cc:439] T dfc4af3152e446fcb8c1259adff6fc21 P 554b70db09f344a2b8056e8d8bd6bdad: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fd00917592c4e1e91b1e32810acd65a" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 33943 } } peers { permanent_uuid: "554b70db09f344a2b8056e8d8bd6bdad" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 35599 } } peers { permanent_uuid: "f935a73e27894272897d653e030f40b1" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 43747 } } }
I20260810 04:06:43.356564 1887 tablet_replica.cc:442] T dfc4af3152e446fcb8c1259adff6fc21 P 554b70db09f344a2b8056e8d8bd6bdad: This TxnStatusTablet replica's current role is: FOLLOWER
W20260810 04:06:43.390419 1854 tablet.cc:2369] T 7b18c69319c44be59588c14519944b4d P f935a73e27894272897d653e030f40b1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 04:06:43.636257 1886 raft_consensus.cc:482] T dfc4af3152e446fcb8c1259adff6fc21 P f935a73e27894272897d653e030f40b1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 04:06:43.636351 1886 raft_consensus.cc:504] T dfc4af3152e446fcb8c1259adff6fc21 P f935a73e27894272897d653e030f40b1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fd00917592c4e1e91b1e32810acd65a" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 33943 } } peers { permanent_uuid: "554b70db09f344a2b8056e8d8bd6bdad" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 35599 } } peers { permanent_uuid: "f935a73e27894272897d653e030f40b1" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 43747 } }
I20260810 04:06:43.636497 1886 leader_election.cc:288] T dfc4af3152e446fcb8c1259adff6fc21 P f935a73e27894272897d653e030f40b1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fd00917592c4e1e91b1e32810acd65a (127.31.32.1:33943), 554b70db09f344a2b8056e8d8bd6bdad (127.31.32.2:35599)
I20260810 04:06:43.636790 1484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfc4af3152e446fcb8c1259adff6fc21" candidate_uuid: "f935a73e27894272897d653e030f40b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fd00917592c4e1e91b1e32810acd65a" is_pre_election: true
I20260810 04:06:43.636881 1484 raft_consensus.cc:2454] T dfc4af3152e446fcb8c1259adff6fc21 P 3fd00917592c4e1e91b1e32810acd65a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f935a73e27894272897d653e030f40b1 in term 0.
I20260810 04:06:43.636879 1650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfc4af3152e446fcb8c1259adff6fc21" candidate_uuid: "f935a73e27894272897d653e030f40b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "554b70db09f344a2b8056e8d8bd6bdad" is_pre_election: true
I20260810 04:06:43.637050 1650 raft_consensus.cc:2454] T dfc4af3152e446fcb8c1259adff6fc21 P 554b70db09f344a2b8056e8d8bd6bdad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f935a73e27894272897d653e030f40b1 in term 0.
I20260810 04:06:43.637076 1732 leader_election.cc:302] T dfc4af3152e446fcb8c1259adff6fc21 P f935a73e27894272897d653e030f40b1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fd00917592c4e1e91b1e32810acd65a, f935a73e27894272897d653e030f40b1; no voters:
I20260810 04:06:43.637182 1886 raft_consensus.cc:2781] T dfc4af3152e446fcb8c1259adff6fc21 P f935a73e27894272897d653e030f40b1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 04:06:43.637266 1886 raft_consensus.cc:482] T dfc4af3152e446fcb8c1259adff6fc21 P f935a73e27894272897d653e030f40b1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 04:06:43.637317 1886 raft_consensus.cc:3037] T dfc4af3152e446fcb8c1259adff6fc21 P f935a73e27894272897d653e030f40b1 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:06:43.638049 1886 raft_consensus.cc:504] T dfc4af3152e446fcb8c1259adff6fc21 P f935a73e27894272897d653e030f40b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fd00917592c4e1e91b1e32810acd65a" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 33943 } } peers { permanent_uuid: "554b70db09f344a2b8056e8d8bd6bdad" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 35599 } } peers { permanent_uuid: "f935a73e27894272897d653e030f40b1" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 43747 } }
I20260810 04:06:43.638206 1886 leader_election.cc:288] T dfc4af3152e446fcb8c1259adff6fc21 P f935a73e27894272897d653e030f40b1 [CANDIDATE]: Term 1 election: Requested vote from peers 3fd00917592c4e1e91b1e32810acd65a (127.31.32.1:33943), 554b70db09f344a2b8056e8d8bd6bdad (127.31.32.2:35599)
I20260810 04:06:43.638406 1484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfc4af3152e446fcb8c1259adff6fc21" candidate_uuid: "f935a73e27894272897d653e030f40b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fd00917592c4e1e91b1e32810acd65a"
I20260810 04:06:43.638427 1650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfc4af3152e446fcb8c1259adff6fc21" candidate_uuid: "f935a73e27894272897d653e030f40b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "554b70db09f344a2b8056e8d8bd6bdad"
I20260810 04:06:43.638497 1650 raft_consensus.cc:3037] T dfc4af3152e446fcb8c1259adff6fc21 P 554b70db09f344a2b8056e8d8bd6bdad [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:06:43.638507 1484 raft_consensus.cc:3037] T dfc4af3152e446fcb8c1259adff6fc21 P 3fd00917592c4e1e91b1e32810acd65a [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:06:43.639144 1484 raft_consensus.cc:2454] T dfc4af3152e446fcb8c1259adff6fc21 P 3fd00917592c4e1e91b1e32810acd65a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f935a73e27894272897d653e030f40b1 in term 1.
I20260810 04:06:43.639160 1650 raft_consensus.cc:2454] T dfc4af3152e446fcb8c1259adff6fc21 P 554b70db09f344a2b8056e8d8bd6bdad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f935a73e27894272897d653e030f40b1 in term 1.
I20260810 04:06:43.639334 1732 leader_election.cc:302] T dfc4af3152e446fcb8c1259adff6fc21 P f935a73e27894272897d653e030f40b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 554b70db09f344a2b8056e8d8bd6bdad, f935a73e27894272897d653e030f40b1; no voters:
I20260810 04:06:43.639458 1886 raft_consensus.cc:2781] T dfc4af3152e446fcb8c1259adff6fc21 P f935a73e27894272897d653e030f40b1 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:06:43.639539 1886 raft_consensus.cc:686] T dfc4af3152e446fcb8c1259adff6fc21 P f935a73e27894272897d653e030f40b1 [term 1 LEADER]: Becoming Leader. State: Replica: f935a73e27894272897d653e030f40b1, State: Running, Role: LEADER
I20260810 04:06:43.639595 1886 consensus_queue.cc:237] T dfc4af3152e446fcb8c1259adff6fc21 P f935a73e27894272897d653e030f40b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fd00917592c4e1e91b1e32810acd65a" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 33943 } } peers { permanent_uuid: "554b70db09f344a2b8056e8d8bd6bdad" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 35599 } } peers { permanent_uuid: "f935a73e27894272897d653e030f40b1" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 43747 } }
I20260810 04:06:43.639925 1889 tablet_replica.cc:439] T dfc4af3152e446fcb8c1259adff6fc21 P f935a73e27894272897d653e030f40b1: TxnStatusTablet state changed. Reason: New leader f935a73e27894272897d653e030f40b1. Latest consensus state: current_term: 1 leader_uuid: "f935a73e27894272897d653e030f40b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fd00917592c4e1e91b1e32810acd65a" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 33943 } } peers { permanent_uuid: "554b70db09f344a2b8056e8d8bd6bdad" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 35599 } } peers { permanent_uuid: "f935a73e27894272897d653e030f40b1" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 43747 } } }
I20260810 04:06:43.640035 1889 tablet_replica.cc:442] T dfc4af3152e446fcb8c1259adff6fc21 P f935a73e27894272897d653e030f40b1: This TxnStatusTablet replica's current role is: LEADER
I20260810 04:06:43.640297 1901 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260810 04:06:43.640291 1325 catalog_manager.cc:5495] T dfc4af3152e446fcb8c1259adff6fc21 P f935a73e27894272897d653e030f40b1 reported cstate change: term changed from 0 to 1, leader changed from <none> to f935a73e27894272897d653e030f40b1 (127.31.32.3). New cstate: current_term: 1 leader_uuid: "f935a73e27894272897d653e030f40b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fd00917592c4e1e91b1e32810acd65a" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 33943 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "554b70db09f344a2b8056e8d8bd6bdad" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 35599 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f935a73e27894272897d653e030f40b1" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 43747 } health_report { overall_health: HEALTHY } } }
I20260810 04:06:43.640453 1901 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260810 04:06:43.708911 1904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:43.706218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35024 (local address 127.31.32.3:43747)
0810 04:06:43.706316 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0810 04:06:43.706320 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 04:06:43.706424 (+ 104us) server_negotiation.cc:407] Connection header received
0810 04:06:43.706524 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:06:43.706525 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:06:43.706550 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:06:43.706632 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 04:06:43.706761 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:43.707258 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:43.707982 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:43.708245 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:43.708320 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:06:43.708530 (+ 210us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 04:06:43.708628 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 04:06:43.708709 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0810 04:06:43.708768 (+ 59us) server_negotiation.cc:299] Negotiation successful
0810 04:06:43.708796 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75,"thread_start_us":48,"threads_started":1}
I20260810 04:06:43.708899 1896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:43.706165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.3:43747 (local address 127.0.0.1:35024)
0810 04:06:43.706361 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0810 04:06:43.706372 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 04:06:43.706444 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:06:43.706637 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:06:43.706638 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:06:43.706640 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 04:06:43.706698 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:06:43.706700 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:06:43.707344 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:06:43.707345 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:06:43.707900 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:06:43.707901 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:06:43.708322 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:06:43.708322 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:06:43.708389 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:06:43.708409 (+ 20us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 04:06:43.708703 (+ 294us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 04:06:43.708713 (+ 10us) client_negotiation.cc:769] Sending connection context
0810 04:06:43.708763 (+ 50us) client_negotiation.cc:240] Negotiation successful
0810 04:06:43.708812 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156}
I20260810 04:06:43.710659 1650 raft_consensus.cc:1261] T dfc4af3152e446fcb8c1259adff6fc21 P 554b70db09f344a2b8056e8d8bd6bdad [term 1 FOLLOWER]: Refusing update from remote peer f935a73e27894272897d653e030f40b1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:06:43.710732 1484 raft_consensus.cc:1261] T dfc4af3152e446fcb8c1259adff6fc21 P 3fd00917592c4e1e91b1e32810acd65a [term 1 FOLLOWER]: Refusing update from remote peer f935a73e27894272897d653e030f40b1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:06:43.710997 1889 consensus_queue.cc:1035] T dfc4af3152e446fcb8c1259adff6fc21 P f935a73e27894272897d653e030f40b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "554b70db09f344a2b8056e8d8bd6bdad" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 35599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:06:43.711109 1886 consensus_queue.cc:1035] T dfc4af3152e446fcb8c1259adff6fc21 P f935a73e27894272897d653e030f40b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fd00917592c4e1e91b1e32810acd65a" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 33943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:06:43.712186 1887 tablet_replica.cc:439] T dfc4af3152e446fcb8c1259adff6fc21 P 554b70db09f344a2b8056e8d8bd6bdad: TxnStatusTablet state changed. Reason: New leader f935a73e27894272897d653e030f40b1. Latest consensus state: current_term: 1 leader_uuid: "f935a73e27894272897d653e030f40b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fd00917592c4e1e91b1e32810acd65a" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 33943 } } peers { permanent_uuid: "554b70db09f344a2b8056e8d8bd6bdad" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 35599 } } peers { permanent_uuid: "f935a73e27894272897d653e030f40b1" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 43747 } } }
I20260810 04:06:43.712282 1887 tablet_replica.cc:442] T dfc4af3152e446fcb8c1259adff6fc21 P 554b70db09f344a2b8056e8d8bd6bdad: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 04:06:43.712815 1885 tablet_replica.cc:439] T dfc4af3152e446fcb8c1259adff6fc21 P 3fd00917592c4e1e91b1e32810acd65a: TxnStatusTablet state changed. Reason: New leader f935a73e27894272897d653e030f40b1. Latest consensus state: current_term: 1 leader_uuid: "f935a73e27894272897d653e030f40b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fd00917592c4e1e91b1e32810acd65a" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 33943 } } peers { permanent_uuid: "554b70db09f344a2b8056e8d8bd6bdad" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 35599 } } peers { permanent_uuid: "f935a73e27894272897d653e030f40b1" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 43747 } } }
I20260810 04:06:43.712914 1885 tablet_replica.cc:442] T dfc4af3152e446fcb8c1259adff6fc21 P 3fd00917592c4e1e91b1e32810acd65a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 04:06:43.713709 1889 tablet_replica.cc:439] T dfc4af3152e446fcb8c1259adff6fc21 P f935a73e27894272897d653e030f40b1: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f935a73e27894272897d653e030f40b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fd00917592c4e1e91b1e32810acd65a" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 33943 } } peers { permanent_uuid: "554b70db09f344a2b8056e8d8bd6bdad" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 35599 } } peers { permanent_uuid: "f935a73e27894272897d653e030f40b1" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 43747 } } }
I20260810 04:06:43.713794 1889 tablet_replica.cc:442] T dfc4af3152e446fcb8c1259adff6fc21 P f935a73e27894272897d653e030f40b1: This TxnStatusTablet replica's current role is: LEADER
I20260810 04:06:43.713845 1889 tablet_replica.cc:439] T dfc4af3152e446fcb8c1259adff6fc21 P f935a73e27894272897d653e030f40b1: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f935a73e27894272897d653e030f40b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fd00917592c4e1e91b1e32810acd65a" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 33943 } } peers { permanent_uuid: "554b70db09f344a2b8056e8d8bd6bdad" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 35599 } } peers { permanent_uuid: "f935a73e27894272897d653e030f40b1" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 43747 } } }
I20260810 04:06:43.713851 1907 mvcc.cc:205] Tried to move back new op lower bound from 7316827355996262400 to 7316827355708268544. Current Snapshot: MvccSnapshot[applied={T|T < 7316827355996262400}]
I20260810 04:06:43.713893 1889 tablet_replica.cc:442] T dfc4af3152e446fcb8c1259adff6fc21 P f935a73e27894272897d653e030f40b1: This TxnStatusTablet replica's current role is: LEADER
I20260810 04:06:43.713950 1905 mvcc.cc:205] Tried to move back new op lower bound from 7316827355996262400 to 7316827355708268544. Current Snapshot: MvccSnapshot[applied={T|T < 7316827355996262400}]
I20260810 04:06:43.714367 1887 tablet_replica.cc:439] T dfc4af3152e446fcb8c1259adff6fc21 P 554b70db09f344a2b8056e8d8bd6bdad: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f935a73e27894272897d653e030f40b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fd00917592c4e1e91b1e32810acd65a" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 33943 } } peers { permanent_uuid: "554b70db09f344a2b8056e8d8bd6bdad" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 35599 } } peers { permanent_uuid: "f935a73e27894272897d653e030f40b1" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 43747 } } }
I20260810 04:06:43.714453 1887 tablet_replica.cc:442] T dfc4af3152e446fcb8c1259adff6fc21 P 554b70db09f344a2b8056e8d8bd6bdad: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 04:06:43.714510 1885 tablet_replica.cc:439] T dfc4af3152e446fcb8c1259adff6fc21 P 3fd00917592c4e1e91b1e32810acd65a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f935a73e27894272897d653e030f40b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fd00917592c4e1e91b1e32810acd65a" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 33943 } } peers { permanent_uuid: "554b70db09f344a2b8056e8d8bd6bdad" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 35599 } } peers { permanent_uuid: "f935a73e27894272897d653e030f40b1" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 43747 } } }
I20260810 04:06:43.714591 1885 tablet_replica.cc:442] T dfc4af3152e446fcb8c1259adff6fc21 P 3fd00917592c4e1e91b1e32810acd65a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 04:06:43.718737 1904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:43.716186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35032 (local address 127.31.32.3:43747)
0810 04:06:43.716244 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0810 04:06:43.716246 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:06:43.716372 (+ 126us) server_negotiation.cc:407] Connection header received
0810 04:06:43.716435 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:06:43.716436 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:06:43.716444 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:06:43.716488 (+ 44us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 04:06:43.716612 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:43.717032 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:43.718115 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:43.718309 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:43.718354 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:06:43.718539 (+ 185us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 04:06:43.718562 (+ 23us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 04:06:43.718632 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0810 04:06:43.718647 (+ 15us) server_negotiation.cc:299] Negotiation successful
0810 04:06:43.718670 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260810 04:06:43.723800 1904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:43.721356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.3:60965 (local address 127.31.32.3:43747)
0810 04:06:43.721394 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0810 04:06:43.721397 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:06:43.721531 (+ 134us) server_negotiation.cc:407] Connection header received
0810 04:06:43.721575 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:06:43.721576 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:06:43.721584 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:06:43.721633 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 04:06:43.721758 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:43.722153 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:43.723201 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:43.723370 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:43.723412 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:06:43.723612 (+ 200us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 04:06:43.723633 (+ 21us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 04:06:43.723665 (+ 32us) server_negotiation.cc:1035] Waiting for connection context
0810 04:06:43.723726 (+ 61us) server_negotiation.cc:299] Negotiation successful
0810 04:06:43.723740 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260810 04:06:43.723958 1921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:43.721319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.3:43747 (local address 127.31.32.3:60965)
0810 04:06:43.721454 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0810 04:06:43.721465 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 04:06:43.721527 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:06:43.721638 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:06:43.721639 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:06:43.721641 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 04:06:43.721702 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:06:43.721703 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:06:43.722227 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:06:43.722228 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:06:43.723106 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:06:43.723109 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:06:43.723438 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:06:43.723439 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:06:43.723506 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:06:43.723521 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 04:06:43.723678 (+ 157us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 04:06:43.723680 (+ 2us) client_negotiation.cc:769] Sending connection context
0810 04:06:43.723725 (+ 45us) client_negotiation.cc:240] Negotiation successful
0810 04:06:43.723763 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":74,"threads_started":1}
I20260810 04:06:43.726536 1650 raft_consensus.cc:1261] T 7b18c69319c44be59588c14519944b4d P 554b70db09f344a2b8056e8d8bd6bdad [term 1 FOLLOWER]: Refusing update from remote peer f935a73e27894272897d653e030f40b1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:06:43.726569 1484 raft_consensus.cc:1261] T 7b18c69319c44be59588c14519944b4d P 3fd00917592c4e1e91b1e32810acd65a [term 1 FOLLOWER]: Refusing update from remote peer f935a73e27894272897d653e030f40b1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:06:43.726742 1889 consensus_queue.cc:1035] T 7b18c69319c44be59588c14519944b4d P f935a73e27894272897d653e030f40b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "554b70db09f344a2b8056e8d8bd6bdad" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 35599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:06:43.726854 1886 consensus_queue.cc:1035] T 7b18c69319c44be59588c14519944b4d P f935a73e27894272897d653e030f40b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fd00917592c4e1e91b1e32810acd65a" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 33943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:06:43.728318 1906 mvcc.cc:205] Tried to move back new op lower bound from 7316827356062728192 to 7316827354410295296. Current Snapshot: MvccSnapshot[applied={T|T < 7316827356062728192}]
I20260810 04:06:43.741832 1326 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44238:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 04:06:43.741930 1326 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44238:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 04:06:43.742738 1326 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6fa0f90143354e5bb2c4cbac1d334676: Sending DeleteTablet for 3 replicas of tablet 7b18c69319c44be59588c14519944b4d
I20260810 04:06:43.743043 1476 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b18c69319c44be59588c14519944b4d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:06:43 UTC) from {username='kudu', principal='kudu/127.31.32.62@KRBTEST.COM'} at 127.0.0.1:44436
I20260810 04:06:43.743144 1781 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b18c69319c44be59588c14519944b4d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:06:43 UTC) from {username='kudu', principal='kudu/127.31.32.62@KRBTEST.COM'} at 127.0.0.1:57182
I20260810 04:06:43.743332 1629 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b18c69319c44be59588c14519944b4d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:06:43 UTC) from {username='kudu', principal='kudu/127.31.32.62@KRBTEST.COM'} at 127.0.0.1:41030
I20260810 04:06:43.743497 1926 tablet_replica.cc:330] T 7b18c69319c44be59588c14519944b4d P f935a73e27894272897d653e030f40b1: stopping tablet replica
I20260810 04:06:43.743546 1925 tablet_replica.cc:330] T 7b18c69319c44be59588c14519944b4d P 3fd00917592c4e1e91b1e32810acd65a: stopping tablet replica
I20260810 04:06:43.743598 1926 raft_consensus.cc:2229] T 7b18c69319c44be59588c14519944b4d P f935a73e27894272897d653e030f40b1 [term 1 LEADER]: Raft consensus shutting down.
I20260810 04:06:43.743686 1926 raft_consensus.cc:2258] T 7b18c69319c44be59588c14519944b4d P f935a73e27894272897d653e030f40b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:06:43.743731 1925 raft_consensus.cc:2229] T 7b18c69319c44be59588c14519944b4d P 3fd00917592c4e1e91b1e32810acd65a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 04:06:43.743872 1925 raft_consensus.cc:2258] T 7b18c69319c44be59588c14519944b4d P 3fd00917592c4e1e91b1e32810acd65a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:06:43.743881 1326 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44238:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 04:06:43.744016 1326 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 04:06:43.744366 1925 ts_tablet_manager.cc:1905] T 7b18c69319c44be59588c14519944b4d P 3fd00917592c4e1e91b1e32810acd65a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:06:43.745479 1781 tablet_service.cc:1467] Processing CreateTablet for tablet 324d3d3e3bed4ef9aa78359b2e8eeb67 (DEFAULT_TABLE table=test-table [id=4ebd4bc5cc5b4b648f521f1b3426a82b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:06:43.745602 1781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 324d3d3e3bed4ef9aa78359b2e8eeb67. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:06:43.746317 1925 ts_tablet_manager.cc:1918] T 7b18c69319c44be59588c14519944b4d P 3fd00917592c4e1e91b1e32810acd65a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 04:06:43.746379 1925 log.cc:1196] T 7b18c69319c44be59588c14519944b4d P 3fd00917592c4e1e91b1e32810acd65a: Deleting WAL directory at /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786334794676589-31872-0/minicluster-data/ts-0/wal/wals/7b18c69319c44be59588c14519944b4d
I20260810 04:06:43.746637 1925 ts_tablet_manager.cc:1939] T 7b18c69319c44be59588c14519944b4d P 3fd00917592c4e1e91b1e32810acd65a: Deleting consensus metadata
I20260810 04:06:43.747169 1879 tablet_bootstrap.cc:492] T 324d3d3e3bed4ef9aa78359b2e8eeb67 P f935a73e27894272897d653e030f40b1: Bootstrap starting.
I20260810 04:06:43.747517 1879 tablet_bootstrap.cc:654] T 324d3d3e3bed4ef9aa78359b2e8eeb67 P f935a73e27894272897d653e030f40b1: Neither blocks nor log segments found. Creating new log.
I20260810 04:06:43.747998 1311 catalog_manager.cc:4841] TS 3fd00917592c4e1e91b1e32810acd65a (127.31.32.1:33943): tablet 7b18c69319c44be59588c14519944b4d (table test-table [id=215df5b718154f86aca79b4c9874801a]) successfully deleted
I20260810 04:06:43.748049 1879 tablet_bootstrap.cc:492] T 324d3d3e3bed4ef9aa78359b2e8eeb67 P f935a73e27894272897d653e030f40b1: No bootstrap required, opened a new log
I20260810 04:06:43.748088 1879 ts_tablet_manager.cc:1397] T 324d3d3e3bed4ef9aa78359b2e8eeb67 P f935a73e27894272897d653e030f40b1: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260810 04:06:43.748234 1629 tablet_service.cc:1467] Processing CreateTablet for tablet 324d3d3e3bed4ef9aa78359b2e8eeb67 (DEFAULT_TABLE table=test-table [id=4ebd4bc5cc5b4b648f521f1b3426a82b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:06:43.748243 1879 raft_consensus.cc:348] T 324d3d3e3bed4ef9aa78359b2e8eeb67 P f935a73e27894272897d653e030f40b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fd00917592c4e1e91b1e32810acd65a" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 33943 } } peers { permanent_uuid: "554b70db09f344a2b8056e8d8bd6bdad" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 35599 } } peers { permanent_uuid: "f935a73e27894272897d653e030f40b1" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 43747 } }
I20260810 04:06:43.748306 1879 raft_consensus.cc:374] T 324d3d3e3bed4ef9aa78359b2e8eeb67 P f935a73e27894272897d653e030f40b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:06:43.748328 1879 raft_consensus.cc:729] T 324d3d3e3bed4ef9aa78359b2e8eeb67 P f935a73e27894272897d653e030f40b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f935a73e27894272897d653e030f40b1, State: Initialized, Role: FOLLOWER
I20260810 04:06:43.748349 1629 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 324d3d3e3bed4ef9aa78359b2e8eeb67. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:06:43.748370 1879 consensus_queue.cc:260] T 324d3d3e3bed4ef9aa78359b2e8eeb67 P f935a73e27894272897d653e030f40b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fd00917592c4e1e91b1e32810acd65a" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 33943 } } peers { permanent_uuid: "554b70db09f344a2b8056e8d8bd6bdad" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 35599 } } peers { permanent_uuid: "f935a73e27894272897d653e030f40b1" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 43747 } }
I20260810 04:06:43.748457 1879 ts_tablet_manager.cc:1428] T 324d3d3e3bed4ef9aa78359b2e8eeb67 P f935a73e27894272897d653e030f40b1: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260810 04:06:43.749379 1926 ts_tablet_manager.cc:1905] T 7b18c69319c44be59588c14519944b4d P f935a73e27894272897d653e030f40b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:06:43.749782 1880 tablet_bootstrap.cc:492] T 324d3d3e3bed4ef9aa78359b2e8eeb67 P 554b70db09f344a2b8056e8d8bd6bdad: Bootstrap starting.
I20260810 04:06:43.749933 1476 tablet_service.cc:1467] Processing CreateTablet for tablet 324d3d3e3bed4ef9aa78359b2e8eeb67 (DEFAULT_TABLE table=test-table [id=4ebd4bc5cc5b4b648f521f1b3426a82b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:06:43.750032 1476 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 324d3d3e3bed4ef9aa78359b2e8eeb67. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:06:43.750448 1880 tablet_bootstrap.cc:654] T 324d3d3e3bed4ef9aa78359b2e8eeb67 P 554b70db09f344a2b8056e8d8bd6bdad: Neither blocks nor log segments found. Creating new log.
I20260810 04:06:43.751134 1926 ts_tablet_manager.cc:1918] T 7b18c69319c44be59588c14519944b4d P f935a73e27894272897d653e030f40b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 04:06:43.751215 1926 log.cc:1196] T 7b18c69319c44be59588c14519944b4d P f935a73e27894272897d653e030f40b1: Deleting WAL directory at /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786334794676589-31872-0/minicluster-data/ts-2/wal/wals/7b18c69319c44be59588c14519944b4d
I20260810 04:06:43.751420 1880 tablet_bootstrap.cc:492] T 324d3d3e3bed4ef9aa78359b2e8eeb67 P 554b70db09f344a2b8056e8d8bd6bdad: No bootstrap required, opened a new log
I20260810 04:06:43.751420 1878 tablet_bootstrap.cc:492] T 324d3d3e3bed4ef9aa78359b2e8eeb67 P 3fd00917592c4e1e91b1e32810acd65a: Bootstrap starting.
I20260810 04:06:43.751464 1880 ts_tablet_manager.cc:1397] T 324d3d3e3bed4ef9aa78359b2e8eeb67 P 554b70db09f344a2b8056e8d8bd6bdad: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 04:06:43.751513 1926 ts_tablet_manager.cc:1939] T 7b18c69319c44be59588c14519944b4d P f935a73e27894272897d653e030f40b1: Deleting consensus metadata
I20260810 04:06:43.751693 1880 raft_consensus.cc:348] T 324d3d3e3bed4ef9aa78359b2e8eeb67 P 554b70db09f344a2b8056e8d8bd6bdad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fd00917592c4e1e91b1e32810acd65a" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 33943 } } peers { permanent_uuid: "554b70db09f344a2b8056e8d8bd6bdad" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 35599 } } peers { permanent_uuid: "f935a73e27894272897d653e030f40b1" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 43747 } }
I20260810 04:06:43.751837 1878 tablet_bootstrap.cc:654] T 324d3d3e3bed4ef9aa78359b2e8eeb67 P 3fd00917592c4e1e91b1e32810acd65a: Neither blocks nor log segments found. Creating new log.
I20260810 04:06:43.751812 1880 raft_consensus.cc:374] T 324d3d3e3bed4ef9aa78359b2e8eeb67 P 554b70db09f344a2b8056e8d8bd6bdad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:06:43.751873 1880 raft_consensus.cc:729] T 324d3d3e3bed4ef9aa78359b2e8eeb67 P 554b70db09f344a2b8056e8d8bd6bdad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 554b70db09f344a2b8056e8d8bd6bdad, State: Initialized, Role: FOLLOWER
I20260810 04:06:43.751950 1880 consensus_queue.cc:260] T 324d3d3e3bed4ef9aa78359b2e8eeb67 P 554b70db09f344a2b8056e8d8bd6bdad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fd00917592c4e1e91b1e32810acd65a" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 33943 } } peers { permanent_uuid: "554b70db09f344a2b8056e8d8bd6bdad" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 35599 } } peers { permanent_uuid: "f935a73e27894272897d653e030f40b1" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 43747 } }
I20260810 04:06:43.752077 1880 ts_tablet_manager.cc:1428] T 324d3d3e3bed4ef9aa78359b2e8eeb67 P 554b70db09f344a2b8056e8d8bd6bdad: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260810 04:06:43.752199 1310 catalog_manager.cc:4841] TS f935a73e27894272897d653e030f40b1 (127.31.32.3:43747): tablet 7b18c69319c44be59588c14519944b4d (table test-table [id=215df5b718154f86aca79b4c9874801a]) successfully deleted
I20260810 04:06:43.753696 1927 tablet_replica.cc:330] T 7b18c69319c44be59588c14519944b4d P 554b70db09f344a2b8056e8d8bd6bdad: stopping tablet replica
I20260810 04:06:43.753835 1878 tablet_bootstrap.cc:492] T 324d3d3e3bed4ef9aa78359b2e8eeb67 P 3fd00917592c4e1e91b1e32810acd65a: No bootstrap required, opened a new log
I20260810 04:06:43.753859 1927 raft_consensus.cc:2229] T 7b18c69319c44be59588c14519944b4d P 554b70db09f344a2b8056e8d8bd6bdad [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 04:06:43.753913 1927 raft_consensus.cc:2258] T 7b18c69319c44be59588c14519944b4d P 554b70db09f344a2b8056e8d8bd6bdad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:06:43.753908 1878 ts_tablet_manager.cc:1397] T 324d3d3e3bed4ef9aa78359b2e8eeb67 P 3fd00917592c4e1e91b1e32810acd65a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 04:06:43.754362 1927 ts_tablet_manager.cc:1905] T 7b18c69319c44be59588c14519944b4d P 554b70db09f344a2b8056e8d8bd6bdad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:06:43.754657 1878 raft_consensus.cc:348] T 324d3d3e3bed4ef9aa78359b2e8eeb67 P 3fd00917592c4e1e91b1e32810acd65a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fd00917592c4e1e91b1e32810acd65a" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 33943 } } peers { permanent_uuid: "554b70db09f344a2b8056e8d8bd6bdad" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 35599 } } peers { permanent_uuid: "f935a73e27894272897d653e030f40b1" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 43747 } }
I20260810 04:06:43.754793 1878 raft_consensus.cc:374] T 324d3d3e3bed4ef9aa78359b2e8eeb67 P 3fd00917592c4e1e91b1e32810acd65a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:06:43.754819 1878 raft_consensus.cc:729] T 324d3d3e3bed4ef9aa78359b2e8eeb67 P 3fd00917592c4e1e91b1e32810acd65a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fd00917592c4e1e91b1e32810acd65a, State: Initialized, Role: FOLLOWER
I20260810 04:06:43.754866 1878 consensus_queue.cc:260] T 324d3d3e3bed4ef9aa78359b2e8eeb67 P 3fd00917592c4e1e91b1e32810acd65a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fd00917592c4e1e91b1e32810acd65a" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 33943 } } peers { permanent_uuid: "554b70db09f344a2b8056e8d8bd6bdad" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 35599 } } peers { permanent_uuid: "f935a73e27894272897d653e030f40b1" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 43747 } }
I20260810 04:06:43.754976 1878 ts_tablet_manager.cc:1428] T 324d3d3e3bed4ef9aa78359b2e8eeb67 P 3fd00917592c4e1e91b1e32810acd65a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260810 04:06:43.755908 1927 ts_tablet_manager.cc:1918] T 7b18c69319c44be59588c14519944b4d P 554b70db09f344a2b8056e8d8bd6bdad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 04:06:43.756057 1927 log.cc:1196] T 7b18c69319c44be59588c14519944b4d P 554b70db09f344a2b8056e8d8bd6bdad: Deleting WAL directory at /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786334794676589-31872-0/minicluster-data/ts-1/wal/wals/7b18c69319c44be59588c14519944b4d
I20260810 04:06:43.756335 1927 ts_tablet_manager.cc:1939] T 7b18c69319c44be59588c14519944b4d P 554b70db09f344a2b8056e8d8bd6bdad: Deleting consensus metadata
I20260810 04:06:43.756743 1311 catalog_manager.cc:4841] TS 554b70db09f344a2b8056e8d8bd6bdad (127.31.32.2:35599): tablet 7b18c69319c44be59588c14519944b4d (table test-table [id=215df5b718154f86aca79b4c9874801a]) successfully deleted
I20260810 04:06:43.764742 1886 raft_consensus.cc:482] T 324d3d3e3bed4ef9aa78359b2e8eeb67 P f935a73e27894272897d653e030f40b1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 04:06:43.764819 1886 raft_consensus.cc:504] T 324d3d3e3bed4ef9aa78359b2e8eeb67 P f935a73e27894272897d653e030f40b1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fd00917592c4e1e91b1e32810acd65a" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 33943 } } peers { permanent_uuid: "554b70db09f344a2b8056e8d8bd6bdad" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 35599 } } peers { permanent_uuid: "f935a73e27894272897d653e030f40b1" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 43747 } }
I20260810 04:06:43.764945 1886 leader_election.cc:288] T 324d3d3e3bed4ef9aa78359b2e8eeb67 P f935a73e27894272897d653e030f40b1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fd00917592c4e1e91b1e32810acd65a (127.31.32.1:33943), 554b70db09f344a2b8056e8d8bd6bdad (127.31.32.2:35599)
I20260810 04:06:43.765203 1650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "324d3d3e3bed4ef9aa78359b2e8eeb67" candidate_uuid: "f935a73e27894272897d653e030f40b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "554b70db09f344a2b8056e8d8bd6bdad" is_pre_election: true
I20260810 04:06:43.765199 1484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "324d3d3e3bed4ef9aa78359b2e8eeb67" candidate_uuid: "f935a73e27894272897d653e030f40b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fd00917592c4e1e91b1e32810acd65a" is_pre_election: true
I20260810 04:06:43.765300 1484 raft_consensus.cc:2454] T 324d3d3e3bed4ef9aa78359b2e8eeb67 P 3fd00917592c4e1e91b1e32810acd65a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f935a73e27894272897d653e030f40b1 in term 0.
I20260810 04:06:43.765296 1650 raft_consensus.cc:2454] T 324d3d3e3bed4ef9aa78359b2e8eeb67 P 554b70db09f344a2b8056e8d8bd6bdad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f935a73e27894272897d653e030f40b1 in term 0.
I20260810 04:06:43.765466 1732 leader_election.cc:302] T 324d3d3e3bed4ef9aa78359b2e8eeb67 P f935a73e27894272897d653e030f40b1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 554b70db09f344a2b8056e8d8bd6bdad, f935a73e27894272897d653e030f40b1; no voters:
I20260810 04:06:43.765594 1886 raft_consensus.cc:2781] T 324d3d3e3bed4ef9aa78359b2e8eeb67 P f935a73e27894272897d653e030f40b1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 04:06:43.765648 1886 raft_consensus.cc:482] T 324d3d3e3bed4ef9aa78359b2e8eeb67 P f935a73e27894272897d653e030f40b1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 04:06:43.765697 1886 raft_consensus.cc:3037] T 324d3d3e3bed4ef9aa78359b2e8eeb67 P f935a73e27894272897d653e030f40b1 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:06:43.766258 1886 raft_consensus.cc:504] T 324d3d3e3bed4ef9aa78359b2e8eeb67 P f935a73e27894272897d653e030f40b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fd00917592c4e1e91b1e32810acd65a" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 33943 } } peers { permanent_uuid: "554b70db09f344a2b8056e8d8bd6bdad" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 35599 } } peers { permanent_uuid: "f935a73e27894272897d653e030f40b1" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 43747 } }
I20260810 04:06:43.766396 1886 leader_election.cc:288] T 324d3d3e3bed4ef9aa78359b2e8eeb67 P f935a73e27894272897d653e030f40b1 [CANDIDATE]: Term 1 election: Requested vote from peers 3fd00917592c4e1e91b1e32810acd65a (127.31.32.1:33943), 554b70db09f344a2b8056e8d8bd6bdad (127.31.32.2:35599)
I20260810 04:06:43.766556 1650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "324d3d3e3bed4ef9aa78359b2e8eeb67" candidate_uuid: "f935a73e27894272897d653e030f40b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "554b70db09f344a2b8056e8d8bd6bdad"
I20260810 04:06:43.766566 1484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "324d3d3e3bed4ef9aa78359b2e8eeb67" candidate_uuid: "f935a73e27894272897d653e030f40b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fd00917592c4e1e91b1e32810acd65a"
I20260810 04:06:43.766635 1484 raft_consensus.cc:3037] T 324d3d3e3bed4ef9aa78359b2e8eeb67 P 3fd00917592c4e1e91b1e32810acd65a [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:06:43.766636 1650 raft_consensus.cc:3037] T 324d3d3e3bed4ef9aa78359b2e8eeb67 P 554b70db09f344a2b8056e8d8bd6bdad [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:06:43.767158 1484 raft_consensus.cc:2454] T 324d3d3e3bed4ef9aa78359b2e8eeb67 P 3fd00917592c4e1e91b1e32810acd65a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f935a73e27894272897d653e030f40b1 in term 1.
I20260810 04:06:43.767333 1650 raft_consensus.cc:2454] T 324d3d3e3bed4ef9aa78359b2e8eeb67 P 554b70db09f344a2b8056e8d8bd6bdad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f935a73e27894272897d653e030f40b1 in term 1.
I20260810 04:06:43.767359 1732 leader_election.cc:302] T 324d3d3e3bed4ef9aa78359b2e8eeb67 P f935a73e27894272897d653e030f40b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fd00917592c4e1e91b1e32810acd65a, f935a73e27894272897d653e030f40b1; no voters:
I20260810 04:06:43.767457 1886 raft_consensus.cc:2781] T 324d3d3e3bed4ef9aa78359b2e8eeb67 P f935a73e27894272897d653e030f40b1 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:06:43.767498 1886 raft_consensus.cc:686] T 324d3d3e3bed4ef9aa78359b2e8eeb67 P f935a73e27894272897d653e030f40b1 [term 1 LEADER]: Becoming Leader. State: Replica: f935a73e27894272897d653e030f40b1, State: Running, Role: LEADER
I20260810 04:06:43.767551 1886 consensus_queue.cc:237] T 324d3d3e3bed4ef9aa78359b2e8eeb67 P f935a73e27894272897d653e030f40b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fd00917592c4e1e91b1e32810acd65a" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 33943 } } peers { permanent_uuid: "554b70db09f344a2b8056e8d8bd6bdad" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 35599 } } peers { permanent_uuid: "f935a73e27894272897d653e030f40b1" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 43747 } }
I20260810 04:06:43.768198 1326 catalog_manager.cc:5495] T 324d3d3e3bed4ef9aa78359b2e8eeb67 P f935a73e27894272897d653e030f40b1 reported cstate change: term changed from 0 to 1, leader changed from <none> to f935a73e27894272897d653e030f40b1 (127.31.32.3). New cstate: current_term: 1 leader_uuid: "f935a73e27894272897d653e030f40b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fd00917592c4e1e91b1e32810acd65a" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 33943 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "554b70db09f344a2b8056e8d8bd6bdad" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 35599 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f935a73e27894272897d653e030f40b1" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 43747 } health_report { overall_health: HEALTHY } } }
I20260810 04:06:43.777074 1650 raft_consensus.cc:1261] T 324d3d3e3bed4ef9aa78359b2e8eeb67 P 554b70db09f344a2b8056e8d8bd6bdad [term 1 FOLLOWER]: Refusing update from remote peer f935a73e27894272897d653e030f40b1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:06:43.777073 1484 raft_consensus.cc:1261] T 324d3d3e3bed4ef9aa78359b2e8eeb67 P 3fd00917592c4e1e91b1e32810acd65a [term 1 FOLLOWER]: Refusing update from remote peer f935a73e27894272897d653e030f40b1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:06:43.777336 1912 consensus_queue.cc:1035] T 324d3d3e3bed4ef9aa78359b2e8eeb67 P f935a73e27894272897d653e030f40b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fd00917592c4e1e91b1e32810acd65a" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 33943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:06:43.777470 1912 consensus_queue.cc:1035] T 324d3d3e3bed4ef9aa78359b2e8eeb67 P f935a73e27894272897d653e030f40b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "554b70db09f344a2b8056e8d8bd6bdad" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 35599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260810 04:06:43.781584 1905 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260810 04:06:45.299913 1325 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44238:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 04:06:45.300034 1325 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44238:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 04:06:45.300872 1325 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6fa0f90143354e5bb2c4cbac1d334676: Sending DeleteTablet for 3 replicas of tablet 324d3d3e3bed4ef9aa78359b2e8eeb67
I20260810 04:06:45.301147 1476 tablet_service.cc:1514] Processing DeleteTablet for tablet 324d3d3e3bed4ef9aa78359b2e8eeb67 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:06:45 UTC) from {username='kudu', principal='kudu/127.31.32.62@KRBTEST.COM'} at 127.0.0.1:44436
I20260810 04:06:45.301164 1629 tablet_service.cc:1514] Processing DeleteTablet for tablet 324d3d3e3bed4ef9aa78359b2e8eeb67 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:06:45 UTC) from {username='kudu', principal='kudu/127.31.32.62@KRBTEST.COM'} at 127.0.0.1:41030
I20260810 04:06:45.301155 1781 tablet_service.cc:1514] Processing DeleteTablet for tablet 324d3d3e3bed4ef9aa78359b2e8eeb67 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:06:45 UTC) from {username='kudu', principal='kudu/127.31.32.62@KRBTEST.COM'} at 127.0.0.1:57182
I20260810 04:06:45.301373 1958 tablet_replica.cc:330] T 324d3d3e3bed4ef9aa78359b2e8eeb67 P f935a73e27894272897d653e030f40b1: stopping tablet replica
I20260810 04:06:45.301494 1958 raft_consensus.cc:2229] T 324d3d3e3bed4ef9aa78359b2e8eeb67 P f935a73e27894272897d653e030f40b1 [term 1 LEADER]: Raft consensus shutting down.
I20260810 04:06:45.301568 1958 raft_consensus.cc:2258] T 324d3d3e3bed4ef9aa78359b2e8eeb67 P f935a73e27894272897d653e030f40b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:06:45.301813 1958 ts_tablet_manager.cc:1905] T 324d3d3e3bed4ef9aa78359b2e8eeb67 P f935a73e27894272897d653e030f40b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:06:45.303823 1958 ts_tablet_manager.cc:1918] T 324d3d3e3bed4ef9aa78359b2e8eeb67 P f935a73e27894272897d653e030f40b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 04:06:45.303905 1958 log.cc:1196] T 324d3d3e3bed4ef9aa78359b2e8eeb67 P f935a73e27894272897d653e030f40b1: Deleting WAL directory at /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786334794676589-31872-0/minicluster-data/ts-2/wal/wals/324d3d3e3bed4ef9aa78359b2e8eeb67
I20260810 04:06:45.304196 1958 ts_tablet_manager.cc:1939] T 324d3d3e3bed4ef9aa78359b2e8eeb67 P f935a73e27894272897d653e030f40b1: Deleting consensus metadata
I20260810 04:06:45.304353 1955 tablet_replica.cc:330] T 324d3d3e3bed4ef9aa78359b2e8eeb67 P 3fd00917592c4e1e91b1e32810acd65a: stopping tablet replica
I20260810 04:06:45.304420 1955 raft_consensus.cc:2229] T 324d3d3e3bed4ef9aa78359b2e8eeb67 P 3fd00917592c4e1e91b1e32810acd65a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 04:06:45.304471 1955 raft_consensus.cc:2258] T 324d3d3e3bed4ef9aa78359b2e8eeb67 P 3fd00917592c4e1e91b1e32810acd65a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:06:45.304517 1310 catalog_manager.cc:4841] TS f935a73e27894272897d653e030f40b1 (127.31.32.3:43747): tablet 324d3d3e3bed4ef9aa78359b2e8eeb67 (table test-table [id=4ebd4bc5cc5b4b648f521f1b3426a82b]) successfully deleted
I20260810 04:06:45.304581 1956 tablet_replica.cc:330] T 324d3d3e3bed4ef9aa78359b2e8eeb67 P 554b70db09f344a2b8056e8d8bd6bdad: stopping tablet replica
I20260810 04:06:45.304668 1956 raft_consensus.cc:2229] T 324d3d3e3bed4ef9aa78359b2e8eeb67 P 554b70db09f344a2b8056e8d8bd6bdad [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 04:06:45.304731 1956 raft_consensus.cc:2258] T 324d3d3e3bed4ef9aa78359b2e8eeb67 P 554b70db09f344a2b8056e8d8bd6bdad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:06:45.304734 1955 ts_tablet_manager.cc:1905] T 324d3d3e3bed4ef9aa78359b2e8eeb67 P 3fd00917592c4e1e91b1e32810acd65a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:06:45.304975 1956 ts_tablet_manager.cc:1905] T 324d3d3e3bed4ef9aa78359b2e8eeb67 P 554b70db09f344a2b8056e8d8bd6bdad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:06:45.306234 1956 ts_tablet_manager.cc:1918] T 324d3d3e3bed4ef9aa78359b2e8eeb67 P 554b70db09f344a2b8056e8d8bd6bdad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 04:06:45.306342 1956 log.cc:1196] T 324d3d3e3bed4ef9aa78359b2e8eeb67 P 554b70db09f344a2b8056e8d8bd6bdad: Deleting WAL directory at /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786334794676589-31872-0/minicluster-data/ts-1/wal/wals/324d3d3e3bed4ef9aa78359b2e8eeb67
I20260810 04:06:45.306565 1956 ts_tablet_manager.cc:1939] T 324d3d3e3bed4ef9aa78359b2e8eeb67 P 554b70db09f344a2b8056e8d8bd6bdad: Deleting consensus metadata
I20260810 04:06:45.306645 1955 ts_tablet_manager.cc:1918] T 324d3d3e3bed4ef9aa78359b2e8eeb67 P 3fd00917592c4e1e91b1e32810acd65a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 04:06:45.306715 1955 log.cc:1196] T 324d3d3e3bed4ef9aa78359b2e8eeb67 P 3fd00917592c4e1e91b1e32810acd65a: Deleting WAL directory at /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786334794676589-31872-0/minicluster-data/ts-0/wal/wals/324d3d3e3bed4ef9aa78359b2e8eeb67
I20260810 04:06:45.306872 1311 catalog_manager.cc:4841] TS 554b70db09f344a2b8056e8d8bd6bdad (127.31.32.2:35599): tablet 324d3d3e3bed4ef9aa78359b2e8eeb67 (table test-table [id=4ebd4bc5cc5b4b648f521f1b3426a82b]) successfully deleted
I20260810 04:06:45.306977 1955 ts_tablet_manager.cc:1939] T 324d3d3e3bed4ef9aa78359b2e8eeb67 P 3fd00917592c4e1e91b1e32810acd65a: Deleting consensus metadata
I20260810 04:06:45.307309 1311 catalog_manager.cc:4841] TS 3fd00917592c4e1e91b1e32810acd65a (127.31.32.1:33943): tablet 324d3d3e3bed4ef9aa78359b2e8eeb67 (table test-table [id=4ebd4bc5cc5b4b648f521f1b3426a82b]) successfully deleted
Aug 10 04:06:45 dist-test-slave-8w9v krb5kdc[1277](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334805, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 10 04:06:45 dist-test-slave-8w9v krb5kdc[1277](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334805, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.31.32.62@KRBTEST.COM
I20260810 04:06:45.326103 1966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:45.318487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39006 (local address 127.31.32.62:34751)
0810 04:06:45.318638 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0810 04:06:45.318641 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:06:45.318651 (+ 10us) server_negotiation.cc:407] Connection header received
0810 04:06:45.318727 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:06:45.318728 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:06:45.318747 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:06:45.318831 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:06:45.319424 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:45.319961 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:45.320822 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:45.321035 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:45.321127 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:06:45.323411 (+ 2284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:06:45.323413 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:06:45.323414 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:06:45.323421 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:06:45.324774 (+ 1353us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:06:45.325179 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:06:45.325181 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:06:45.325181 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:06:45.325254 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:06:45.325478 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:06:45.325479 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:06:45.325480 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:06:45.325600 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:06:45.325716 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0810 04:06:45.325865 (+ 149us) server_negotiation.cc:299] Negotiation successful
0810 04:06:45.325986 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":73,"threads_started":1}
I20260810 04:06:45.327555 1325 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39006:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 04:06:45.327709 1325 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 04:06:45.329103 1629 tablet_service.cc:1467] Processing CreateTablet for tablet cfeac835a22a4f618bf1ca2348551a11 (DEFAULT_TABLE table=test-table [id=86a90e442f594cdf9837245f41fdf1c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:06:45.329146 1781 tablet_service.cc:1467] Processing CreateTablet for tablet cfeac835a22a4f618bf1ca2348551a11 (DEFAULT_TABLE table=test-table [id=86a90e442f594cdf9837245f41fdf1c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:06:45.329224 1629 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cfeac835a22a4f618bf1ca2348551a11. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:06:45.329257 1781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cfeac835a22a4f618bf1ca2348551a11. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:06:45.330592 1476 tablet_service.cc:1467] Processing CreateTablet for tablet cfeac835a22a4f618bf1ca2348551a11 (DEFAULT_TABLE table=test-table [id=86a90e442f594cdf9837245f41fdf1c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:06:45.330654 1968 tablet_bootstrap.cc:492] T cfeac835a22a4f618bf1ca2348551a11 P f935a73e27894272897d653e030f40b1: Bootstrap starting.
I20260810 04:06:45.330714 1476 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cfeac835a22a4f618bf1ca2348551a11. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:06:45.330991 1968 tablet_bootstrap.cc:654] T cfeac835a22a4f618bf1ca2348551a11 P f935a73e27894272897d653e030f40b1: Neither blocks nor log segments found. Creating new log.
I20260810 04:06:45.331641 1968 tablet_bootstrap.cc:492] T cfeac835a22a4f618bf1ca2348551a11 P f935a73e27894272897d653e030f40b1: No bootstrap required, opened a new log
I20260810 04:06:45.331715 1968 ts_tablet_manager.cc:1397] T cfeac835a22a4f618bf1ca2348551a11 P f935a73e27894272897d653e030f40b1: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260810 04:06:45.331785 1970 tablet_bootstrap.cc:492] T cfeac835a22a4f618bf1ca2348551a11 P 3fd00917592c4e1e91b1e32810acd65a: Bootstrap starting.
I20260810 04:06:45.331804 1967 tablet_bootstrap.cc:492] T cfeac835a22a4f618bf1ca2348551a11 P 554b70db09f344a2b8056e8d8bd6bdad: Bootstrap starting.
I20260810 04:06:45.331835 1968 raft_consensus.cc:348] T cfeac835a22a4f618bf1ca2348551a11 P f935a73e27894272897d653e030f40b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fd00917592c4e1e91b1e32810acd65a" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 33943 } } peers { permanent_uuid: "554b70db09f344a2b8056e8d8bd6bdad" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 35599 } } peers { permanent_uuid: "f935a73e27894272897d653e030f40b1" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 43747 } }
I20260810 04:06:45.331928 1968 raft_consensus.cc:374] T cfeac835a22a4f618bf1ca2348551a11 P f935a73e27894272897d653e030f40b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:06:45.331957 1968 raft_consensus.cc:729] T cfeac835a22a4f618bf1ca2348551a11 P f935a73e27894272897d653e030f40b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f935a73e27894272897d653e030f40b1, State: Initialized, Role: FOLLOWER
I20260810 04:06:45.332027 1968 consensus_queue.cc:260] T cfeac835a22a4f618bf1ca2348551a11 P f935a73e27894272897d653e030f40b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fd00917592c4e1e91b1e32810acd65a" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 33943 } } peers { permanent_uuid: "554b70db09f344a2b8056e8d8bd6bdad" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 35599 } } peers { permanent_uuid: "f935a73e27894272897d653e030f40b1" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 43747 } }
I20260810 04:06:45.332193 1967 tablet_bootstrap.cc:654] T cfeac835a22a4f618bf1ca2348551a11 P 554b70db09f344a2b8056e8d8bd6bdad: Neither blocks nor log segments found. Creating new log.
I20260810 04:06:45.332231 1968 ts_tablet_manager.cc:1428] T cfeac835a22a4f618bf1ca2348551a11 P f935a73e27894272897d653e030f40b1: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260810 04:06:45.332108 1970 tablet_bootstrap.cc:654] T cfeac835a22a4f618bf1ca2348551a11 P 3fd00917592c4e1e91b1e32810acd65a: Neither blocks nor log segments found. Creating new log.
I20260810 04:06:45.332868 1967 tablet_bootstrap.cc:492] T cfeac835a22a4f618bf1ca2348551a11 P 554b70db09f344a2b8056e8d8bd6bdad: No bootstrap required, opened a new log
I20260810 04:06:45.332928 1967 ts_tablet_manager.cc:1397] T cfeac835a22a4f618bf1ca2348551a11 P 554b70db09f344a2b8056e8d8bd6bdad: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260810 04:06:45.333079 1967 raft_consensus.cc:348] T cfeac835a22a4f618bf1ca2348551a11 P 554b70db09f344a2b8056e8d8bd6bdad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fd00917592c4e1e91b1e32810acd65a" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 33943 } } peers { permanent_uuid: "554b70db09f344a2b8056e8d8bd6bdad" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 35599 } } peers { permanent_uuid: "f935a73e27894272897d653e030f40b1" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 43747 } }
I20260810 04:06:45.333164 1967 raft_consensus.cc:374] T cfeac835a22a4f618bf1ca2348551a11 P 554b70db09f344a2b8056e8d8bd6bdad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:06:45.333195 1967 raft_consensus.cc:729] T cfeac835a22a4f618bf1ca2348551a11 P 554b70db09f344a2b8056e8d8bd6bdad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 554b70db09f344a2b8056e8d8bd6bdad, State: Initialized, Role: FOLLOWER
I20260810 04:06:45.333218 1970 tablet_bootstrap.cc:492] T cfeac835a22a4f618bf1ca2348551a11 P 3fd00917592c4e1e91b1e32810acd65a: No bootstrap required, opened a new log
I20260810 04:06:45.333243 1967 consensus_queue.cc:260] T cfeac835a22a4f618bf1ca2348551a11 P 554b70db09f344a2b8056e8d8bd6bdad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fd00917592c4e1e91b1e32810acd65a" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 33943 } } peers { permanent_uuid: "554b70db09f344a2b8056e8d8bd6bdad" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 35599 } } peers { permanent_uuid: "f935a73e27894272897d653e030f40b1" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 43747 } }
I20260810 04:06:45.333277 1970 ts_tablet_manager.cc:1397] T cfeac835a22a4f618bf1ca2348551a11 P 3fd00917592c4e1e91b1e32810acd65a: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260810 04:06:45.333396 1967 ts_tablet_manager.cc:1428] T cfeac835a22a4f618bf1ca2348551a11 P 554b70db09f344a2b8056e8d8bd6bdad: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260810 04:06:45.333446 1970 raft_consensus.cc:348] T cfeac835a22a4f618bf1ca2348551a11 P 3fd00917592c4e1e91b1e32810acd65a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fd00917592c4e1e91b1e32810acd65a" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 33943 } } peers { permanent_uuid: "554b70db09f344a2b8056e8d8bd6bdad" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 35599 } } peers { permanent_uuid: "f935a73e27894272897d653e030f40b1" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 43747 } }
I20260810 04:06:45.333527 1970 raft_consensus.cc:374] T cfeac835a22a4f618bf1ca2348551a11 P 3fd00917592c4e1e91b1e32810acd65a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:06:45.333555 1970 raft_consensus.cc:729] T cfeac835a22a4f618bf1ca2348551a11 P 3fd00917592c4e1e91b1e32810acd65a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fd00917592c4e1e91b1e32810acd65a, State: Initialized, Role: FOLLOWER
I20260810 04:06:45.333631 1970 consensus_queue.cc:260] T cfeac835a22a4f618bf1ca2348551a11 P 3fd00917592c4e1e91b1e32810acd65a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fd00917592c4e1e91b1e32810acd65a" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 33943 } } peers { permanent_uuid: "554b70db09f344a2b8056e8d8bd6bdad" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 35599 } } peers { permanent_uuid: "f935a73e27894272897d653e030f40b1" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 43747 } }
I20260810 04:06:45.333822 1970 ts_tablet_manager.cc:1428] T cfeac835a22a4f618bf1ca2348551a11 P 3fd00917592c4e1e91b1e32810acd65a: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260810 04:06:45.350584 1974 raft_consensus.cc:482] T cfeac835a22a4f618bf1ca2348551a11 P 554b70db09f344a2b8056e8d8bd6bdad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 04:06:45.350660 1974 raft_consensus.cc:504] T cfeac835a22a4f618bf1ca2348551a11 P 554b70db09f344a2b8056e8d8bd6bdad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fd00917592c4e1e91b1e32810acd65a" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 33943 } } peers { permanent_uuid: "554b70db09f344a2b8056e8d8bd6bdad" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 35599 } } peers { permanent_uuid: "f935a73e27894272897d653e030f40b1" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 43747 } }
I20260810 04:06:45.350993 1974 leader_election.cc:288] T cfeac835a22a4f618bf1ca2348551a11 P 554b70db09f344a2b8056e8d8bd6bdad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fd00917592c4e1e91b1e32810acd65a (127.31.32.1:33943), f935a73e27894272897d653e030f40b1 (127.31.32.3:43747)
I20260810 04:06:45.354127 1977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:45.351151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.3:43747 (local address 127.31.32.2:45323)
0810 04:06:45.351312 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0810 04:06:45.351324 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 04:06:45.351386 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:06:45.351568 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:06:45.351568 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:06:45.351571 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:06:45.351640 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:06:45.351642 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:06:45.352374 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:06:45.352375 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:06:45.353136 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:06:45.353138 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:06:45.353930 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:06:45.353930 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:06:45.353978 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:06:45.353982 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 04:06:45.354018 (+ 36us) client_negotiation.cc:240] Negotiation successful
0810 04:06:45.354047 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":82,"threads_started":1}
I20260810 04:06:45.354158 1978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:45.351211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.2:45323 (local address 127.31.32.3:43747)
0810 04:06:45.351358 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0810 04:06:45.351360 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:06:45.351375 (+ 15us) server_negotiation.cc:407] Connection header received
0810 04:06:45.351453 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:06:45.351455 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:06:45.351465 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:06:45.351540 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:06:45.351712 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:45.352281 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:45.353221 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:45.353854 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:45.353922 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:06:45.353932 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0810 04:06:45.354040 (+ 108us) server_negotiation.cc:299] Negotiation successful
0810 04:06:45.354076 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":75,"threads_started":1}
I20260810 04:06:45.354354 1803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfeac835a22a4f618bf1ca2348551a11" candidate_uuid: "554b70db09f344a2b8056e8d8bd6bdad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f935a73e27894272897d653e030f40b1" is_pre_election: true
I20260810 04:06:45.354501 1803 raft_consensus.cc:2454] T cfeac835a22a4f618bf1ca2348551a11 P f935a73e27894272897d653e030f40b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 554b70db09f344a2b8056e8d8bd6bdad in term 0.
I20260810 04:06:45.354700 1579 leader_election.cc:302] T cfeac835a22a4f618bf1ca2348551a11 P 554b70db09f344a2b8056e8d8bd6bdad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 554b70db09f344a2b8056e8d8bd6bdad, f935a73e27894272897d653e030f40b1; no voters:
I20260810 04:06:45.354856 1974 raft_consensus.cc:2781] T cfeac835a22a4f618bf1ca2348551a11 P 554b70db09f344a2b8056e8d8bd6bdad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 04:06:45.354907 1974 raft_consensus.cc:482] T cfeac835a22a4f618bf1ca2348551a11 P 554b70db09f344a2b8056e8d8bd6bdad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 04:06:45.354934 1974 raft_consensus.cc:3037] T cfeac835a22a4f618bf1ca2348551a11 P 554b70db09f344a2b8056e8d8bd6bdad [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:06:45.355386 1979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:45.351218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.2:33191 (local address 127.31.32.1:33943)
0810 04:06:45.351346 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0810 04:06:45.351349 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:06:45.351403 (+ 54us) server_negotiation.cc:407] Connection header received
0810 04:06:45.351512 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:06:45.351512 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:06:45.351539 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:06:45.351602 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:06:45.351762 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:45.352305 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:45.354157 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:45.355071 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:45.355127 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:06:45.355134 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0810 04:06:45.355305 (+ 171us) server_negotiation.cc:299] Negotiation successful
0810 04:06:45.355333 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":77,"threads_started":1}
I20260810 04:06:45.355400 1976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:45.351175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.1:33943 (local address 127.31.32.2:33191)
0810 04:06:45.351344 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0810 04:06:45.351355 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 04:06:45.351438 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:06:45.351619 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:06:45.351620 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:06:45.351621 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:06:45.351691 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:06:45.351693 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:06:45.352407 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:06:45.352408 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:06:45.354075 (+ 1667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:06:45.354078 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:06:45.355153 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:06:45.355154 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:06:45.355229 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:06:45.355233 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 04:06:45.355297 (+ 64us) client_negotiation.cc:240] Negotiation successful
0810 04:06:45.355331 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"mutex_wait_us":20,"thread_start_us":92,"threads_started":1}
I20260810 04:06:45.355736 1494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfeac835a22a4f618bf1ca2348551a11" candidate_uuid: "554b70db09f344a2b8056e8d8bd6bdad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fd00917592c4e1e91b1e32810acd65a" is_pre_election: true
I20260810 04:06:45.355839 1494 raft_consensus.cc:2454] T cfeac835a22a4f618bf1ca2348551a11 P 3fd00917592c4e1e91b1e32810acd65a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 554b70db09f344a2b8056e8d8bd6bdad in term 0.
I20260810 04:06:45.355816 1974 raft_consensus.cc:504] T cfeac835a22a4f618bf1ca2348551a11 P 554b70db09f344a2b8056e8d8bd6bdad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fd00917592c4e1e91b1e32810acd65a" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 33943 } } peers { permanent_uuid: "554b70db09f344a2b8056e8d8bd6bdad" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 35599 } } peers { permanent_uuid: "f935a73e27894272897d653e030f40b1" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 43747 } }
I20260810 04:06:45.355927 1974 leader_election.cc:288] T cfeac835a22a4f618bf1ca2348551a11 P 554b70db09f344a2b8056e8d8bd6bdad [CANDIDATE]: Term 1 election: Requested vote from peers 3fd00917592c4e1e91b1e32810acd65a (127.31.32.1:33943), f935a73e27894272897d653e030f40b1 (127.31.32.3:43747)
I20260810 04:06:45.356045 1494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfeac835a22a4f618bf1ca2348551a11" candidate_uuid: "554b70db09f344a2b8056e8d8bd6bdad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fd00917592c4e1e91b1e32810acd65a"
I20260810 04:06:45.356155 1494 raft_consensus.cc:3037] T cfeac835a22a4f618bf1ca2348551a11 P 3fd00917592c4e1e91b1e32810acd65a [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:06:45.356190 1803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfeac835a22a4f618bf1ca2348551a11" candidate_uuid: "554b70db09f344a2b8056e8d8bd6bdad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f935a73e27894272897d653e030f40b1"
I20260810 04:06:45.356248 1803 raft_consensus.cc:3037] T cfeac835a22a4f618bf1ca2348551a11 P f935a73e27894272897d653e030f40b1 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:06:45.356798 1494 raft_consensus.cc:2454] T cfeac835a22a4f618bf1ca2348551a11 P 3fd00917592c4e1e91b1e32810acd65a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 554b70db09f344a2b8056e8d8bd6bdad in term 1.
I20260810 04:06:45.356834 1803 raft_consensus.cc:2454] T cfeac835a22a4f618bf1ca2348551a11 P f935a73e27894272897d653e030f40b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 554b70db09f344a2b8056e8d8bd6bdad in term 1.
I20260810 04:06:45.357013 1580 leader_election.cc:302] T cfeac835a22a4f618bf1ca2348551a11 P 554b70db09f344a2b8056e8d8bd6bdad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fd00917592c4e1e91b1e32810acd65a, 554b70db09f344a2b8056e8d8bd6bdad; no voters:
I20260810 04:06:45.357115 1974 raft_consensus.cc:2781] T cfeac835a22a4f618bf1ca2348551a11 P 554b70db09f344a2b8056e8d8bd6bdad [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:06:45.357260 1974 raft_consensus.cc:686] T cfeac835a22a4f618bf1ca2348551a11 P 554b70db09f344a2b8056e8d8bd6bdad [term 1 LEADER]: Becoming Leader. State: Replica: 554b70db09f344a2b8056e8d8bd6bdad, State: Running, Role: LEADER
I20260810 04:06:45.357326 1974 consensus_queue.cc:237] T cfeac835a22a4f618bf1ca2348551a11 P 554b70db09f344a2b8056e8d8bd6bdad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fd00917592c4e1e91b1e32810acd65a" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 33943 } } peers { permanent_uuid: "554b70db09f344a2b8056e8d8bd6bdad" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 35599 } } peers { permanent_uuid: "f935a73e27894272897d653e030f40b1" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 43747 } }
I20260810 04:06:45.357901 1325 catalog_manager.cc:5495] T cfeac835a22a4f618bf1ca2348551a11 P 554b70db09f344a2b8056e8d8bd6bdad reported cstate change: term changed from 0 to 1, leader changed from <none> to 554b70db09f344a2b8056e8d8bd6bdad (127.31.32.2). New cstate: current_term: 1 leader_uuid: "554b70db09f344a2b8056e8d8bd6bdad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fd00917592c4e1e91b1e32810acd65a" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 33943 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "554b70db09f344a2b8056e8d8bd6bdad" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 35599 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f935a73e27894272897d653e030f40b1" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 43747 } health_report { overall_health: UNKNOWN } } }
I20260810 04:06:45.371317 1983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:45.369018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40614 (local address 127.31.32.2:35599)
0810 04:06:45.369170 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0810 04:06:45.369172 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:06:45.369180 (+ 8us) server_negotiation.cc:407] Connection header received
0810 04:06:45.369221 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:06:45.369222 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:06:45.369238 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:06:45.369313 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 04:06:45.369464 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:45.369924 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:45.370650 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:45.370851 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:45.370904 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:06:45.371035 (+ 131us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 04:06:45.371135 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 04:06:45.371195 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0810 04:06:45.371217 (+ 22us) server_negotiation.cc:299] Negotiation successful
0810 04:06:45.371231 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":103,"threads_started":1}
I20260810 04:06:45.375633 1978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:45.373627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.2:48267 (local address 127.31.32.3:43747)
0810 04:06:45.373683 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0810 04:06:45.373685 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:06:45.373776 (+ 91us) server_negotiation.cc:407] Connection header received
0810 04:06:45.373823 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:06:45.373824 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:06:45.373834 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:06:45.373878 (+ 44us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 04:06:45.373985 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:45.374407 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:45.375071 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:45.375245 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:45.375291 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:06:45.375452 (+ 161us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 04:06:45.375475 (+ 23us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 04:06:45.375511 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0810 04:06:45.375565 (+ 54us) server_negotiation.cc:299] Negotiation successful
0810 04:06:45.375583 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260810 04:06:45.375660 1986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:45.373558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.3:43747 (local address 127.31.32.2:48267)
0810 04:06:45.373717 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0810 04:06:45.373727 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 04:06:45.373779 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:06:45.373897 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:06:45.373898 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:06:45.373899 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 04:06:45.373946 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:06:45.373947 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:06:45.374465 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:06:45.374466 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:06:45.375003 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:06:45.375005 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:06:45.375326 (+ 321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:06:45.375327 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:06:45.375374 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:06:45.375386 (+ 12us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 04:06:45.375529 (+ 143us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 04:06:45.375530 (+ 1us) client_negotiation.cc:769] Sending connection context
0810 04:06:45.375565 (+ 35us) client_negotiation.cc:240] Negotiation successful
0810 04:06:45.375595 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":82,"threads_started":1}
I20260810 04:06:45.377722 1803 raft_consensus.cc:1261] T cfeac835a22a4f618bf1ca2348551a11 P f935a73e27894272897d653e030f40b1 [term 1 FOLLOWER]: Refusing update from remote peer 554b70db09f344a2b8056e8d8bd6bdad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:06:45.377926 1494 raft_consensus.cc:1261] T cfeac835a22a4f618bf1ca2348551a11 P 3fd00917592c4e1e91b1e32810acd65a [term 1 FOLLOWER]: Refusing update from remote peer 554b70db09f344a2b8056e8d8bd6bdad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:06:45.377979 1974 consensus_queue.cc:1035] T cfeac835a22a4f618bf1ca2348551a11 P 554b70db09f344a2b8056e8d8bd6bdad [LEADER]: Connected to new peer: Peer: permanent_uuid: "f935a73e27894272897d653e030f40b1" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 43747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:06:45.378130 1980 consensus_queue.cc:1035] T cfeac835a22a4f618bf1ca2348551a11 P 554b70db09f344a2b8056e8d8bd6bdad [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fd00917592c4e1e91b1e32810acd65a" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 33943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:06:45.385509 1983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:45.383018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.3:50971 (local address 127.31.32.2:35599)
0810 04:06:45.383055 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0810 04:06:45.383057 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:06:45.383215 (+ 158us) server_negotiation.cc:407] Connection header received
0810 04:06:45.383240 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:06:45.383241 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:06:45.383252 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:06:45.383303 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 04:06:45.383439 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:45.383979 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:45.384720 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:45.384981 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:45.385062 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:06:45.385266 (+ 204us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 04:06:45.385296 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 04:06:45.385357 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0810 04:06:45.385430 (+ 73us) server_negotiation.cc:299] Negotiation successful
0810 04:06:45.385449 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260810 04:06:45.385553 1992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:45.382984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.2:35599 (local address 127.31.32.3:50971)
0810 04:06:45.383105 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0810 04:06:45.383115 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 04:06:45.383179 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:06:45.383315 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:06:45.383316 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:06:45.383317 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 04:06:45.383362 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:06:45.383365 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:06:45.384063 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:06:45.384064 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:06:45.384650 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:06:45.384651 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:06:45.385064 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:06:45.385065 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:06:45.385127 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:06:45.385133 (+ 6us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 04:06:45.385371 (+ 238us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 04:06:45.385373 (+ 2us) client_negotiation.cc:769] Sending connection context
0810 04:06:45.385425 (+ 52us) client_negotiation.cc:240] Negotiation successful
0810 04:06:45.385474 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":79,"threads_started":1}
I20260810 04:06:45.392204 1326 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39006:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 04:06:45.392323 1326 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39006:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 04:06:45.392956 1326 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6fa0f90143354e5bb2c4cbac1d334676: Sending DeleteTablet for 3 replicas of tablet cfeac835a22a4f618bf1ca2348551a11
I20260810 04:06:45.393213 1476 tablet_service.cc:1514] Processing DeleteTablet for tablet cfeac835a22a4f618bf1ca2348551a11 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:06:45 UTC) from {username='kudu', principal='kudu/127.31.32.62@KRBTEST.COM'} at 127.0.0.1:44436
I20260810 04:06:45.393289 1629 tablet_service.cc:1514] Processing DeleteTablet for tablet cfeac835a22a4f618bf1ca2348551a11 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:06:45 UTC) from {username='kudu', principal='kudu/127.31.32.62@KRBTEST.COM'} at 127.0.0.1:41030
I20260810 04:06:45.393314 1955 tablet_replica.cc:330] T cfeac835a22a4f618bf1ca2348551a11 P 3fd00917592c4e1e91b1e32810acd65a: stopping tablet replica
I20260810 04:06:45.393286 1781 tablet_service.cc:1514] Processing DeleteTablet for tablet cfeac835a22a4f618bf1ca2348551a11 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:06:45 UTC) from {username='kudu', principal='kudu/127.31.32.62@KRBTEST.COM'} at 127.0.0.1:57182
I20260810 04:06:45.393360 1955 raft_consensus.cc:2229] T cfeac835a22a4f618bf1ca2348551a11 P 3fd00917592c4e1e91b1e32810acd65a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 04:06:45.393373 1956 tablet_replica.cc:330] T cfeac835a22a4f618bf1ca2348551a11 P 554b70db09f344a2b8056e8d8bd6bdad: stopping tablet replica
I20260810 04:06:45.393409 1955 raft_consensus.cc:2258] T cfeac835a22a4f618bf1ca2348551a11 P 3fd00917592c4e1e91b1e32810acd65a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:06:45.393415 1956 raft_consensus.cc:2229] T cfeac835a22a4f618bf1ca2348551a11 P 554b70db09f344a2b8056e8d8bd6bdad [term 1 LEADER]: Raft consensus shutting down.
I20260810 04:06:45.393486 1958 tablet_replica.cc:330] T cfeac835a22a4f618bf1ca2348551a11 P f935a73e27894272897d653e030f40b1: stopping tablet replica
I20260810 04:06:45.393517 1956 raft_consensus.cc:2258] T cfeac835a22a4f618bf1ca2348551a11 P 554b70db09f344a2b8056e8d8bd6bdad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:06:45.393545 1958 raft_consensus.cc:2229] T cfeac835a22a4f618bf1ca2348551a11 P f935a73e27894272897d653e030f40b1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 04:06:45.393577 1958 raft_consensus.cc:2258] T cfeac835a22a4f618bf1ca2348551a11 P f935a73e27894272897d653e030f40b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:06:45.393826 1955 ts_tablet_manager.cc:1905] T cfeac835a22a4f618bf1ca2348551a11 P 3fd00917592c4e1e91b1e32810acd65a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:06:45.393877 1958 ts_tablet_manager.cc:1905] T cfeac835a22a4f618bf1ca2348551a11 P f935a73e27894272897d653e030f40b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:06:45.393953 1956 ts_tablet_manager.cc:1905] T cfeac835a22a4f618bf1ca2348551a11 P 554b70db09f344a2b8056e8d8bd6bdad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:06:45.395572 1958 ts_tablet_manager.cc:1918] T cfeac835a22a4f618bf1ca2348551a11 P f935a73e27894272897d653e030f40b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 04:06:45.395632 1958 log.cc:1196] T cfeac835a22a4f618bf1ca2348551a11 P f935a73e27894272897d653e030f40b1: Deleting WAL directory at /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786334794676589-31872-0/minicluster-data/ts-2/wal/wals/cfeac835a22a4f618bf1ca2348551a11
I20260810 04:06:45.395653 1956 ts_tablet_manager.cc:1918] T cfeac835a22a4f618bf1ca2348551a11 P 554b70db09f344a2b8056e8d8bd6bdad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 04:06:45.395709 1956 log.cc:1196] T cfeac835a22a4f618bf1ca2348551a11 P 554b70db09f344a2b8056e8d8bd6bdad: Deleting WAL directory at /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786334794676589-31872-0/minicluster-data/ts-1/wal/wals/cfeac835a22a4f618bf1ca2348551a11
I20260810 04:06:45.395881 1958 ts_tablet_manager.cc:1939] T cfeac835a22a4f618bf1ca2348551a11 P f935a73e27894272897d653e030f40b1: Deleting consensus metadata
I20260810 04:06:45.395987 1956 ts_tablet_manager.cc:1939] T cfeac835a22a4f618bf1ca2348551a11 P 554b70db09f344a2b8056e8d8bd6bdad: Deleting consensus metadata
I20260810 04:06:45.396016 1326 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39006:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 04:06:45.396183 1326 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 04:06:45.396224 1310 catalog_manager.cc:4841] TS f935a73e27894272897d653e030f40b1 (127.31.32.3:43747): tablet cfeac835a22a4f618bf1ca2348551a11 (table test-table [id=86a90e442f594cdf9837245f41fdf1c0]) successfully deleted
I20260810 04:06:45.396353 1311 catalog_manager.cc:4841] TS 554b70db09f344a2b8056e8d8bd6bdad (127.31.32.2:35599): tablet cfeac835a22a4f618bf1ca2348551a11 (table test-table [id=86a90e442f594cdf9837245f41fdf1c0]) successfully deleted
I20260810 04:06:45.396494 1955 ts_tablet_manager.cc:1918] T cfeac835a22a4f618bf1ca2348551a11 P 3fd00917592c4e1e91b1e32810acd65a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 04:06:45.396543 1955 log.cc:1196] T cfeac835a22a4f618bf1ca2348551a11 P 3fd00917592c4e1e91b1e32810acd65a: Deleting WAL directory at /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786334794676589-31872-0/minicluster-data/ts-0/wal/wals/cfeac835a22a4f618bf1ca2348551a11
I20260810 04:06:45.396759 1955 ts_tablet_manager.cc:1939] T cfeac835a22a4f618bf1ca2348551a11 P 3fd00917592c4e1e91b1e32810acd65a: Deleting consensus metadata
I20260810 04:06:45.397064 1311 catalog_manager.cc:4841] TS 3fd00917592c4e1e91b1e32810acd65a (127.31.32.1:33943): tablet cfeac835a22a4f618bf1ca2348551a11 (table test-table [id=86a90e442f594cdf9837245f41fdf1c0]) successfully deleted
I20260810 04:06:45.397330 1629 tablet_service.cc:1467] Processing CreateTablet for tablet 0e00c45c7be443cbb5888891859165c0 (DEFAULT_TABLE table=test-table [id=0f91a5b4d1694892894be785dbded251]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:06:45.397388 1781 tablet_service.cc:1467] Processing CreateTablet for tablet 0e00c45c7be443cbb5888891859165c0 (DEFAULT_TABLE table=test-table [id=0f91a5b4d1694892894be785dbded251]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:06:45.397390 1476 tablet_service.cc:1467] Processing CreateTablet for tablet 0e00c45c7be443cbb5888891859165c0 (DEFAULT_TABLE table=test-table [id=0f91a5b4d1694892894be785dbded251]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:06:45.397437 1629 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e00c45c7be443cbb5888891859165c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:06:45.397477 1781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e00c45c7be443cbb5888891859165c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:06:45.397496 1476 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e00c45c7be443cbb5888891859165c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:06:45.398604 1967 tablet_bootstrap.cc:492] T 0e00c45c7be443cbb5888891859165c0 P 554b70db09f344a2b8056e8d8bd6bdad: Bootstrap starting.
I20260810 04:06:45.398605 1968 tablet_bootstrap.cc:492] T 0e00c45c7be443cbb5888891859165c0 P f935a73e27894272897d653e030f40b1: Bootstrap starting.
I20260810 04:06:45.398640 1970 tablet_bootstrap.cc:492] T 0e00c45c7be443cbb5888891859165c0 P 3fd00917592c4e1e91b1e32810acd65a: Bootstrap starting.
I20260810 04:06:45.398908 1968 tablet_bootstrap.cc:654] T 0e00c45c7be443cbb5888891859165c0 P f935a73e27894272897d653e030f40b1: Neither blocks nor log segments found. Creating new log.
I20260810 04:06:45.398990 1967 tablet_bootstrap.cc:654] T 0e00c45c7be443cbb5888891859165c0 P 554b70db09f344a2b8056e8d8bd6bdad: Neither blocks nor log segments found. Creating new log.
I20260810 04:06:45.399039 1970 tablet_bootstrap.cc:654] T 0e00c45c7be443cbb5888891859165c0 P 3fd00917592c4e1e91b1e32810acd65a: Neither blocks nor log segments found. Creating new log.
I20260810 04:06:45.399724 1967 tablet_bootstrap.cc:492] T 0e00c45c7be443cbb5888891859165c0 P 554b70db09f344a2b8056e8d8bd6bdad: No bootstrap required, opened a new log
I20260810 04:06:45.399724 1968 tablet_bootstrap.cc:492] T 0e00c45c7be443cbb5888891859165c0 P f935a73e27894272897d653e030f40b1: No bootstrap required, opened a new log
I20260810 04:06:45.399772 1967 ts_tablet_manager.cc:1397] T 0e00c45c7be443cbb5888891859165c0 P 554b70db09f344a2b8056e8d8bd6bdad: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260810 04:06:45.399772 1968 ts_tablet_manager.cc:1397] T 0e00c45c7be443cbb5888891859165c0 P f935a73e27894272897d653e030f40b1: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260810 04:06:45.399883 1968 raft_consensus.cc:348] T 0e00c45c7be443cbb5888891859165c0 P f935a73e27894272897d653e030f40b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fd00917592c4e1e91b1e32810acd65a" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 33943 } } peers { permanent_uuid: "554b70db09f344a2b8056e8d8bd6bdad" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 35599 } } peers { permanent_uuid: "f935a73e27894272897d653e030f40b1" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 43747 } }
I20260810 04:06:45.399950 1968 raft_consensus.cc:374] T 0e00c45c7be443cbb5888891859165c0 P f935a73e27894272897d653e030f40b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:06:45.399953 1970 tablet_bootstrap.cc:492] T 0e00c45c7be443cbb5888891859165c0 P 3fd00917592c4e1e91b1e32810acd65a: No bootstrap required, opened a new log
I20260810 04:06:45.399972 1968 raft_consensus.cc:729] T 0e00c45c7be443cbb5888891859165c0 P f935a73e27894272897d653e030f40b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f935a73e27894272897d653e030f40b1, State: Initialized, Role: FOLLOWER
I20260810 04:06:45.399930 1967 raft_consensus.cc:348] T 0e00c45c7be443cbb5888891859165c0 P 554b70db09f344a2b8056e8d8bd6bdad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fd00917592c4e1e91b1e32810acd65a" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 33943 } } peers { permanent_uuid: "554b70db09f344a2b8056e8d8bd6bdad" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 35599 } } peers { permanent_uuid: "f935a73e27894272897d653e030f40b1" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 43747 } }
I20260810 04:06:45.399996 1970 ts_tablet_manager.cc:1397] T 0e00c45c7be443cbb5888891859165c0 P 3fd00917592c4e1e91b1e32810acd65a: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260810 04:06:45.400000 1967 raft_consensus.cc:374] T 0e00c45c7be443cbb5888891859165c0 P 554b70db09f344a2b8056e8d8bd6bdad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:06:45.400022 1967 raft_consensus.cc:729] T 0e00c45c7be443cbb5888891859165c0 P 554b70db09f344a2b8056e8d8bd6bdad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 554b70db09f344a2b8056e8d8bd6bdad, State: Initialized, Role: FOLLOWER
I20260810 04:06:45.400015 1968 consensus_queue.cc:260] T 0e00c45c7be443cbb5888891859165c0 P f935a73e27894272897d653e030f40b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fd00917592c4e1e91b1e32810acd65a" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 33943 } } peers { permanent_uuid: "554b70db09f344a2b8056e8d8bd6bdad" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 35599 } } peers { permanent_uuid: "f935a73e27894272897d653e030f40b1" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 43747 } }
I20260810 04:06:45.400058 1967 consensus_queue.cc:260] T 0e00c45c7be443cbb5888891859165c0 P 554b70db09f344a2b8056e8d8bd6bdad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fd00917592c4e1e91b1e32810acd65a" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 33943 } } peers { permanent_uuid: "554b70db09f344a2b8056e8d8bd6bdad" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 35599 } } peers { permanent_uuid: "f935a73e27894272897d653e030f40b1" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 43747 } }
I20260810 04:06:45.400175 1967 ts_tablet_manager.cc:1428] T 0e00c45c7be443cbb5888891859165c0 P 554b70db09f344a2b8056e8d8bd6bdad: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260810 04:06:45.400156 1970 raft_consensus.cc:348] T 0e00c45c7be443cbb5888891859165c0 P 3fd00917592c4e1e91b1e32810acd65a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fd00917592c4e1e91b1e32810acd65a" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 33943 } } peers { permanent_uuid: "554b70db09f344a2b8056e8d8bd6bdad" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 35599 } } peers { permanent_uuid: "f935a73e27894272897d653e030f40b1" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 43747 } }
I20260810 04:06:45.400223 1970 raft_consensus.cc:374] T 0e00c45c7be443cbb5888891859165c0 P 3fd00917592c4e1e91b1e32810acd65a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:06:45.400243 1970 raft_consensus.cc:729] T 0e00c45c7be443cbb5888891859165c0 P 3fd00917592c4e1e91b1e32810acd65a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fd00917592c4e1e91b1e32810acd65a, State: Initialized, Role: FOLLOWER
I20260810 04:06:45.400285 1970 consensus_queue.cc:260] T 0e00c45c7be443cbb5888891859165c0 P 3fd00917592c4e1e91b1e32810acd65a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fd00917592c4e1e91b1e32810acd65a" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 33943 } } peers { permanent_uuid: "554b70db09f344a2b8056e8d8bd6bdad" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 35599 } } peers { permanent_uuid: "f935a73e27894272897d653e030f40b1" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 43747 } }
I20260810 04:06:45.400372 1970 ts_tablet_manager.cc:1428] T 0e00c45c7be443cbb5888891859165c0 P 3fd00917592c4e1e91b1e32810acd65a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260810 04:06:45.400112 1968 ts_tablet_manager.cc:1428] T 0e00c45c7be443cbb5888891859165c0 P f935a73e27894272897d653e030f40b1: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260810 04:06:45.469996 1975 raft_consensus.cc:482] T 0e00c45c7be443cbb5888891859165c0 P 3fd00917592c4e1e91b1e32810acd65a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 04:06:45.470131 1975 raft_consensus.cc:504] T 0e00c45c7be443cbb5888891859165c0 P 3fd00917592c4e1e91b1e32810acd65a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fd00917592c4e1e91b1e32810acd65a" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 33943 } } peers { permanent_uuid: "554b70db09f344a2b8056e8d8bd6bdad" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 35599 } } peers { permanent_uuid: "f935a73e27894272897d653e030f40b1" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 43747 } }
I20260810 04:06:45.470534 1975 leader_election.cc:288] T 0e00c45c7be443cbb5888891859165c0 P 3fd00917592c4e1e91b1e32810acd65a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 554b70db09f344a2b8056e8d8bd6bdad (127.31.32.2:35599), f935a73e27894272897d653e030f40b1 (127.31.32.3:43747)
I20260810 04:06:45.474207 1983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:45.470951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.1:46315 (local address 127.31.32.2:35599)
0810 04:06:45.471000 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0810 04:06:45.471001 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 04:06:45.471010 (+ 9us) server_negotiation.cc:407] Connection header received
0810 04:06:45.471037 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:06:45.471037 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:06:45.471046 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:06:45.471094 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:06:45.471342 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:45.472206 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:45.473439 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:45.473836 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:45.473923 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:06:45.473935 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0810 04:06:45.474099 (+ 164us) server_negotiation.cc:299] Negotiation successful
0810 04:06:45.474138 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260810 04:06:45.474225 1997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:45.470739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.2:35599 (local address 127.31.32.1:46315)
0810 04:06:45.470877 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0810 04:06:45.470889 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 04:06:45.470952 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:06:45.471144 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:06:45.471145 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:06:45.471149 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:06:45.471252 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:06:45.471255 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:06:45.472335 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:06:45.472336 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:06:45.473344 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:06:45.473346 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:06:45.473926 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:06:45.473927 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:06:45.474009 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:06:45.474014 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 04:06:45.474078 (+ 64us) client_negotiation.cc:240] Negotiation successful
0810 04:06:45.474128 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":69,"threads_started":1}
I20260810 04:06:45.474490 1650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e00c45c7be443cbb5888891859165c0" candidate_uuid: "3fd00917592c4e1e91b1e32810acd65a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "554b70db09f344a2b8056e8d8bd6bdad" is_pre_election: true
I20260810 04:06:45.474592 1650 raft_consensus.cc:2454] T 0e00c45c7be443cbb5888891859165c0 P 554b70db09f344a2b8056e8d8bd6bdad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fd00917592c4e1e91b1e32810acd65a in term 0.
I20260810 04:06:45.474762 1426 leader_election.cc:302] T 0e00c45c7be443cbb5888891859165c0 P 3fd00917592c4e1e91b1e32810acd65a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fd00917592c4e1e91b1e32810acd65a, 554b70db09f344a2b8056e8d8bd6bdad; no voters:
I20260810 04:06:45.474912 1975 raft_consensus.cc:2781] T 0e00c45c7be443cbb5888891859165c0 P 3fd00917592c4e1e91b1e32810acd65a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 04:06:45.474956 1975 raft_consensus.cc:482] T 0e00c45c7be443cbb5888891859165c0 P 3fd00917592c4e1e91b1e32810acd65a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 04:06:45.474984 1975 raft_consensus.cc:3037] T 0e00c45c7be443cbb5888891859165c0 P 3fd00917592c4e1e91b1e32810acd65a [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:06:45.475423 1978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:45.470889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.1:59183 (local address 127.31.32.3:43747)
0810 04:06:45.470941 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0810 04:06:45.470943 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:06:45.471192 (+ 249us) server_negotiation.cc:407] Connection header received
0810 04:06:45.471239 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:06:45.471240 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:06:45.471249 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:06:45.471333 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:06:45.471992 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:45.472699 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:45.474288 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:45.474933 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:45.475018 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:06:45.475032 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0810 04:06:45.475275 (+ 243us) server_negotiation.cc:299] Negotiation successful
0810 04:06:45.475337 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260810 04:06:45.475469 1998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:45.470739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.3:43747 (local address 127.31.32.1:59183)
0810 04:06:45.471087 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0810 04:06:45.471099 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 04:06:45.471184 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:06:45.471842 (+ 658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:06:45.471843 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:06:45.471846 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:06:45.471922 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:06:45.471924 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:06:45.472878 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:06:45.472879 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:06:45.474105 (+ 1226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:06:45.474108 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:06:45.475074 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:06:45.475076 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:06:45.475173 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:06:45.475178 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 04:06:45.475246 (+ 68us) client_negotiation.cc:240] Negotiation successful
0810 04:06:45.475294 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"mutex_wait_us":27,"thread_start_us":127,"threads_started":1}
I20260810 04:06:45.475850 1803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e00c45c7be443cbb5888891859165c0" candidate_uuid: "3fd00917592c4e1e91b1e32810acd65a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f935a73e27894272897d653e030f40b1" is_pre_election: true
I20260810 04:06:45.475960 1803 raft_consensus.cc:2454] T 0e00c45c7be443cbb5888891859165c0 P f935a73e27894272897d653e030f40b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fd00917592c4e1e91b1e32810acd65a in term 0.
I20260810 04:06:45.476486 1975 raft_consensus.cc:504] T 0e00c45c7be443cbb5888891859165c0 P 3fd00917592c4e1e91b1e32810acd65a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fd00917592c4e1e91b1e32810acd65a" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 33943 } } peers { permanent_uuid: "554b70db09f344a2b8056e8d8bd6bdad" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 35599 } } peers { permanent_uuid: "f935a73e27894272897d653e030f40b1" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 43747 } }
I20260810 04:06:45.476653 1975 leader_election.cc:288] T 0e00c45c7be443cbb5888891859165c0 P 3fd00917592c4e1e91b1e32810acd65a [CANDIDATE]: Term 1 election: Requested vote from peers 554b70db09f344a2b8056e8d8bd6bdad (127.31.32.2:35599), f935a73e27894272897d653e030f40b1 (127.31.32.3:43747)
I20260810 04:06:45.476859 1803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e00c45c7be443cbb5888891859165c0" candidate_uuid: "3fd00917592c4e1e91b1e32810acd65a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f935a73e27894272897d653e030f40b1"
I20260810 04:06:45.476877 1650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e00c45c7be443cbb5888891859165c0" candidate_uuid: "3fd00917592c4e1e91b1e32810acd65a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "554b70db09f344a2b8056e8d8bd6bdad"
I20260810 04:06:45.476963 1650 raft_consensus.cc:3037] T 0e00c45c7be443cbb5888891859165c0 P 554b70db09f344a2b8056e8d8bd6bdad [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:06:45.476969 1803 raft_consensus.cc:3037] T 0e00c45c7be443cbb5888891859165c0 P f935a73e27894272897d653e030f40b1 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:06:45.477602 1803 raft_consensus.cc:2454] T 0e00c45c7be443cbb5888891859165c0 P f935a73e27894272897d653e030f40b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fd00917592c4e1e91b1e32810acd65a in term 1.
I20260810 04:06:45.477785 1425 leader_election.cc:302] T 0e00c45c7be443cbb5888891859165c0 P 3fd00917592c4e1e91b1e32810acd65a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fd00917592c4e1e91b1e32810acd65a, f935a73e27894272897d653e030f40b1; no voters:
I20260810 04:06:45.477895 1975 raft_consensus.cc:2781] T 0e00c45c7be443cbb5888891859165c0 P 3fd00917592c4e1e91b1e32810acd65a [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:06:45.478088 1975 raft_consensus.cc:686] T 0e00c45c7be443cbb5888891859165c0 P 3fd00917592c4e1e91b1e32810acd65a [term 1 LEADER]: Becoming Leader. State: Replica: 3fd00917592c4e1e91b1e32810acd65a, State: Running, Role: LEADER
I20260810 04:06:45.478158 1650 raft_consensus.cc:2454] T 0e00c45c7be443cbb5888891859165c0 P 554b70db09f344a2b8056e8d8bd6bdad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fd00917592c4e1e91b1e32810acd65a in term 1.
I20260810 04:06:45.478163 1975 consensus_queue.cc:237] T 0e00c45c7be443cbb5888891859165c0 P 3fd00917592c4e1e91b1e32810acd65a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fd00917592c4e1e91b1e32810acd65a" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 33943 } } peers { permanent_uuid: "554b70db09f344a2b8056e8d8bd6bdad" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 35599 } } peers { permanent_uuid: "f935a73e27894272897d653e030f40b1" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 43747 } }
I20260810 04:06:45.478876 1325 catalog_manager.cc:5495] T 0e00c45c7be443cbb5888891859165c0 P 3fd00917592c4e1e91b1e32810acd65a reported cstate change: term changed from 0 to 1, leader changed from <none> to 3fd00917592c4e1e91b1e32810acd65a (127.31.32.1). New cstate: current_term: 1 leader_uuid: "3fd00917592c4e1e91b1e32810acd65a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fd00917592c4e1e91b1e32810acd65a" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 33943 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "554b70db09f344a2b8056e8d8bd6bdad" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 35599 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f935a73e27894272897d653e030f40b1" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 43747 } health_report { overall_health: UNKNOWN } } }
I20260810 04:06:45.497596 1979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:45.494530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58792 (local address 127.31.32.1:33943)
0810 04:06:45.494573 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0810 04:06:45.494575 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:06:45.494587 (+ 12us) server_negotiation.cc:407] Connection header received
0810 04:06:45.494631 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:06:45.494632 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:06:45.494659 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:06:45.494777 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 04:06:45.494869 (+ 92us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:45.495644 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:45.496509 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:45.496866 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:45.496956 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:06:45.497136 (+ 180us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 04:06:45.497287 (+ 151us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 04:06:45.497386 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0810 04:06:45.497473 (+ 87us) server_negotiation.cc:299] Negotiation successful
0810 04:06:45.497502 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260810 04:06:45.504581 2005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:45.500771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.3:43747 (local address 127.31.32.1:52007)
0810 04:06:45.501013 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0810 04:06:45.501027 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 04:06:45.501090 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:06:45.501294 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:06:45.501295 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:06:45.501297 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 04:06:45.501459 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:06:45.501461 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:06:45.502588 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:06:45.502588 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:06:45.503366 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:06:45.503369 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:06:45.504030 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:06:45.504031 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:06:45.504090 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:06:45.504106 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 04:06:45.504401 (+ 295us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 04:06:45.504404 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 04:06:45.504452 (+ 48us) client_negotiation.cc:240] Negotiation successful
0810 04:06:45.504491 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":99,"threads_started":1}
I20260810 04:06:45.504642 1978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:45.501030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.1:52007 (local address 127.31.32.3:43747)
0810 04:06:45.501103 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0810 04:06:45.501105 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:06:45.501116 (+ 11us) server_negotiation.cc:407] Connection header received
0810 04:06:45.501163 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:06:45.501164 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:06:45.501200 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:06:45.501275 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 04:06:45.501554 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:45.502459 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:45.503458 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:45.503833 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:45.503936 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:06:45.504240 (+ 304us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 04:06:45.504291 (+ 51us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 04:06:45.504373 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0810 04:06:45.504465 (+ 92us) server_negotiation.cc:299] Negotiation successful
0810 04:06:45.504509 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260810 04:06:45.506876 1803 raft_consensus.cc:1261] T 0e00c45c7be443cbb5888891859165c0 P f935a73e27894272897d653e030f40b1 [term 1 FOLLOWER]: Refusing update from remote peer 3fd00917592c4e1e91b1e32810acd65a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:06:45.506992 1650 raft_consensus.cc:1261] T 0e00c45c7be443cbb5888891859165c0 P 554b70db09f344a2b8056e8d8bd6bdad [term 1 FOLLOWER]: Refusing update from remote peer 3fd00917592c4e1e91b1e32810acd65a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:06:45.507259 1975 consensus_queue.cc:1035] T 0e00c45c7be443cbb5888891859165c0 P 3fd00917592c4e1e91b1e32810acd65a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f935a73e27894272897d653e030f40b1" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 43747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:06:45.507822 2000 consensus_queue.cc:1035] T 0e00c45c7be443cbb5888891859165c0 P 3fd00917592c4e1e91b1e32810acd65a [LEADER]: Connected to new peer: Peer: permanent_uuid: "554b70db09f344a2b8056e8d8bd6bdad" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 35599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:06:45.516592 1979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:45.513592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.3:53647 (local address 127.31.32.1:33943)
0810 04:06:45.513970 (+ 378us) server_negotiation.cc:206] Beginning negotiation
0810 04:06:45.513988 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0810 04:06:45.513997 (+ 9us) server_negotiation.cc:407] Connection header received
0810 04:06:45.514020 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:06:45.514020 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:06:45.514031 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:06:45.514102 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 04:06:45.514219 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:45.514811 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:45.515828 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:45.516110 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:45.516191 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:06:45.516372 (+ 181us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 04:06:45.516405 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 04:06:45.516451 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0810 04:06:45.516508 (+ 57us) server_negotiation.cc:299] Negotiation successful
0810 04:06:45.516527 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349}
I20260810 04:06:45.516606 1992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:45.513511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.1:33943 (local address 127.31.32.3:53647)
0810 04:06:45.513580 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0810 04:06:45.513591 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 04:06:45.513629 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:06:45.514106 (+ 477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:06:45.514106 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:06:45.514107 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 04:06:45.514142 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:06:45.514143 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:06:45.514900 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:06:45.514901 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:06:45.515729 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:06:45.515733 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:06:45.516225 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:06:45.516226 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:06:45.516286 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:06:45.516293 (+ 7us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 04:06:45.516452 (+ 159us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 04:06:45.516455 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 04:06:45.516496 (+ 41us) client_negotiation.cc:240] Negotiation successful
0810 04:06:45.516541 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
W20260810 04:06:45.519555 1940 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260810 04:06:47.041733 1325 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39006:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 04:06:47.041837 1325 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39006:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 04:06:47.042709 1325 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6fa0f90143354e5bb2c4cbac1d334676: Sending DeleteTablet for 3 replicas of tablet 0e00c45c7be443cbb5888891859165c0
I20260810 04:06:47.042997 1476 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e00c45c7be443cbb5888891859165c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:06:47 UTC) from {username='kudu', principal='kudu/127.31.32.62@KRBTEST.COM'} at 127.0.0.1:44436
I20260810 04:06:47.043011 1629 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e00c45c7be443cbb5888891859165c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:06:47 UTC) from {username='kudu', principal='kudu/127.31.32.62@KRBTEST.COM'} at 127.0.0.1:41030
I20260810 04:06:47.043010 1781 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e00c45c7be443cbb5888891859165c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:06:47 UTC) from {username='kudu', principal='kudu/127.31.32.62@KRBTEST.COM'} at 127.0.0.1:57182
I20260810 04:06:47.043202 2034 tablet_replica.cc:330] T 0e00c45c7be443cbb5888891859165c0 P 3fd00917592c4e1e91b1e32810acd65a: stopping tablet replica
I20260810 04:06:47.043224 31872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskws89Io/build/release/bin/kudu with pid 1406
I20260810 04:06:47.045698 2033 tablet_replica.cc:330] T 0e00c45c7be443cbb5888891859165c0 P 554b70db09f344a2b8056e8d8bd6bdad: stopping tablet replica
I20260810 04:06:47.045859 2033 raft_consensus.cc:2229] T 0e00c45c7be443cbb5888891859165c0 P 554b70db09f344a2b8056e8d8bd6bdad [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 04:06:47.045961 2033 raft_consensus.cc:2258] T 0e00c45c7be443cbb5888891859165c0 P 554b70db09f344a2b8056e8d8bd6bdad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:06:47.046284 2033 ts_tablet_manager.cc:1905] T 0e00c45c7be443cbb5888891859165c0 P 554b70db09f344a2b8056e8d8bd6bdad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:06:47.047904 2033 ts_tablet_manager.cc:1918] T 0e00c45c7be443cbb5888891859165c0 P 554b70db09f344a2b8056e8d8bd6bdad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 04:06:47.047986 2033 log.cc:1196] T 0e00c45c7be443cbb5888891859165c0 P 554b70db09f344a2b8056e8d8bd6bdad: Deleting WAL directory at /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786334794676589-31872-0/minicluster-data/ts-1/wal/wals/0e00c45c7be443cbb5888891859165c0
I20260810 04:06:47.048213 2035 tablet_replica.cc:330] T 0e00c45c7be443cbb5888891859165c0 P f935a73e27894272897d653e030f40b1: stopping tablet replica
I20260810 04:06:47.048255 2033 ts_tablet_manager.cc:1939] T 0e00c45c7be443cbb5888891859165c0 P 554b70db09f344a2b8056e8d8bd6bdad: Deleting consensus metadata
I20260810 04:06:47.048292 2035 raft_consensus.cc:2229] T 0e00c45c7be443cbb5888891859165c0 P f935a73e27894272897d653e030f40b1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 04:06:47.048381 2035 raft_consensus.cc:2258] T 0e00c45c7be443cbb5888891859165c0 P f935a73e27894272897d653e030f40b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:06:47.048722 1311 catalog_manager.cc:4841] TS 554b70db09f344a2b8056e8d8bd6bdad (127.31.32.2:35599): tablet 0e00c45c7be443cbb5888891859165c0 (table test-table [id=0f91a5b4d1694892894be785dbded251]) successfully deleted
I20260810 04:06:47.048802 2035 ts_tablet_manager.cc:1905] T 0e00c45c7be443cbb5888891859165c0 P f935a73e27894272897d653e030f40b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:06:47.050695 2035 ts_tablet_manager.cc:1918] T 0e00c45c7be443cbb5888891859165c0 P f935a73e27894272897d653e030f40b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 04:06:47.050765 2035 log.cc:1196] T 0e00c45c7be443cbb5888891859165c0 P f935a73e27894272897d653e030f40b1: Deleting WAL directory at /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786334794676589-31872-0/minicluster-data/ts-2/wal/wals/0e00c45c7be443cbb5888891859165c0
I20260810 04:06:47.050976 2035 ts_tablet_manager.cc:1939] T 0e00c45c7be443cbb5888891859165c0 P f935a73e27894272897d653e030f40b1: Deleting consensus metadata
I20260810 04:06:47.051507 1310 catalog_manager.cc:4841] TS f935a73e27894272897d653e030f40b1 (127.31.32.3:43747): tablet 0e00c45c7be443cbb5888891859165c0 (table test-table [id=0f91a5b4d1694892894be785dbded251]) successfully deleted
W20260810 04:06:47.053633 1311 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.32.1:33943 (error 108)
I20260810 04:06:47.054240 31872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskws89Io/build/release/bin/kudu with pid 1558
I20260810 04:06:47.054251 2038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:47.054026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.1:33943 (local address 127.0.0.1:58806)
0810 04:06:47.054155 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0810 04:06:47.054206 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.1:33943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":65,"threads_started":1}
W20260810 04:06:47.054368 1311 catalog_manager.cc:4584] TS 3fd00917592c4e1e91b1e32810acd65a (127.31.32.1:33943): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0e00c45c7be443cbb5888891859165c0: Network error: Client connection negotiation failed: client connection to 127.31.32.1:33943: connect: Connection refused (error 111)
I20260810 04:06:47.062804 31872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskws89Io/build/release/bin/kudu with pid 1707
I20260810 04:06:47.069886 31872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskws89Io/build/release/bin/kudu with pid 1292
2026-08-10T04:06:47Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (5772 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786334794676589-31872-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 04:06:47 dist-test-slave-8w9v krb5kdc[2041](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 04:06:47 dist-test-slave-8w9v krb5kdc[2041](info): set up 2 sockets
Aug 10 04:06:47 dist-test-slave-8w9v krb5kdc[2041](info): commencing operation
krb5kdc: starting...
W20260810 04:06:48.209216 31872 mini_kdc.cc:121] Time spent starting KDC: real 1.117s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260810 04:06:48.349839 31872 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.141s user 0.001s sys 0.001s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260810 04:06:48.479107 31872 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.129s user 0.001s sys 0.001s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260810 04:06:48.628574 31872 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.149s user 0.000s sys 0.002s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786334794676589-31872-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786334794676589-31872-0/krb5kdc/test-user.keytab.
W20260810 04:06:49.117647 31872 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.489s user 0.000s sys 0.002s
Aug 10 04:06:49 dist-test-slave-8w9v krb5kdc[2041](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334809, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T04:06:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T04:06:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.32.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.32.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.32.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.62.keytab.
Entry for principal kudu/127.31.32.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.62.keytab.
W20260810 04:06:49.252437 31872 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.31.32.62: real 0.112s user 0.001s sys 0.002s
I20260810 04:06:49.252750 31872 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskws89Io/build/release/bin/kudu
/tmp/dist-test-taskws89Io/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786334794676589-31872-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786334794676589-31872-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786334794676589-31872-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786334794676589-31872-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.32.62:36217
--webserver_interface=127.31.32.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.62.keytab
--principal=kudu/127.31.32.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.32.20:39159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.32.62:36217
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786334794676589-31872-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786334794676589-31872-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786334794676589-31872-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:06:49.332646 2055 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:06:49.332840 2055 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:06:49.332876 2055 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:06:49.334012 2055 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:06:49.334074 2055 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:06:49.334100 2055 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:06:49.334127 2055 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:06:49.334151 2055 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:06:49.335528 2055 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.32.20:39159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786334794676589-31872-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786334794676589-31872-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.32.62:36217
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.62.keytab
--principal=kudu/127.31.32.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.32.62:36217
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786334794676589-31872-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.32.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2055
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786334794676589-31872-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:06:49.335794 2055 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:06:49.336021 2055 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:06:49.341080 2061 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:06:49.341167 2055 server_base.cc:1034] running on GCE node
W20260810 04:06:49.341094 2062 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:06:49.341094 2064 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:06:49.341560 2055 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:06:49.341840 2055 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:06:49.343001 2055 hybrid_clock.cc:648] HybridClock initialized: now 1786334809342989 us; error 43 us; skew 500 ppm
Aug 10 04:06:49 dist-test-slave-8w9v krb5kdc[2041](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334809, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:06:49.348798 2055 init.cc:377] Logged in from keytab as kudu/127.31.32.62@KRBTEST.COM (short username kudu)
I20260810 04:06:49.349263 2055 webserver.cc:460] Webserver started at http://127.31.32.62:33545/ using document root <none> and password file <none>
I20260810 04:06:49.349517 2055 fs_manager.cc:359] Metadata directory not provided
I20260810 04:06:49.349589 2055 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:06:49.349737 2055 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:06:49.350764 2055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786334794676589-31872-0/minicluster-data/master-0/data/instance:
uuid: "9117a0f1bdef4827a49e0a71bb21dfb3"
format_stamp: "Formatted at 2026-08-10 04:06:49 on dist-test-slave-8w9v"
server_key: "3caf3f889e5f5b61f976e751cd77740f"
server_key_iv: "baef45a1f34f673262c6ccf51fe64778"
server_key_version: "encryptionkey@0"
I20260810 04:06:49.351131 2055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786334794676589-31872-0/minicluster-data/master-0/wal/instance:
uuid: "9117a0f1bdef4827a49e0a71bb21dfb3"
format_stamp: "Formatted at 2026-08-10 04:06:49 on dist-test-slave-8w9v"
server_key: "3caf3f889e5f5b61f976e751cd77740f"
server_key_iv: "baef45a1f34f673262c6ccf51fe64778"
server_key_version: "encryptionkey@0"
I20260810 04:06:49.352502 2055 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 04:06:49.353181 2071 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:06:49.353343 2055 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 04:06:49.353447 2055 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786334794676589-31872-0/minicluster-data/master-0/data,/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786334794676589-31872-0/minicluster-data/master-0/wal
uuid: "9117a0f1bdef4827a49e0a71bb21dfb3"
format_stamp: "Formatted at 2026-08-10 04:06:49 on dist-test-slave-8w9v"
server_key: "3caf3f889e5f5b61f976e751cd77740f"
server_key_iv: "baef45a1f34f673262c6ccf51fe64778"
server_key_version: "encryptionkey@0"
I20260810 04:06:49.353547 2055 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786334794676589-31872-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786334794676589-31872-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786334794676589-31872-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:06:49.380511 2055 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:06:49.382246 2055 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:06:49.382414 2055 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:06:49.386432 2055 rpc_server.cc:307] RPC server started. Bound to: 127.31.32.62:36217
I20260810 04:06:49.386464 2129 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.32.62:36217 every 8 connection(s)
I20260810 04:06:49.386817 2055 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786334794676589-31872-0/minicluster-data/master-0/data/info.pb
I20260810 04:06:49.387708 31872 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskws89Io/build/release/bin/kudu as pid 2055
I20260810 04:06:49.387804 31872 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786334794676589-31872-0/minicluster-data/master-0/wal/instance
I20260810 04:06:49.387804 2130 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:06:49.387998 31872 external_mini_cluster.cc:1428] Setting key 168515a2b475714bd35ccd7be75d5e25
I20260810 04:06:49.391078 2130 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9117a0f1bdef4827a49e0a71bb21dfb3: Bootstrap starting.
I20260810 04:06:49.391752 2130 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9117a0f1bdef4827a49e0a71bb21dfb3: Neither blocks nor log segments found. Creating new log.
I20260810 04:06:49.392066 2130 log.cc:824] T 00000000000000000000000000000000 P 9117a0f1bdef4827a49e0a71bb21dfb3: Log is configured to *not* fsync() on all Append() calls
I20260810 04:06:49.392889 2130 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9117a0f1bdef4827a49e0a71bb21dfb3: No bootstrap required, opened a new log
Aug 10 04:06:49 dist-test-slave-8w9v krb5kdc[2041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334809, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.32.62@KRBTEST.COM
I20260810 04:06:49.395356 2130 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9117a0f1bdef4827a49e0a71bb21dfb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9117a0f1bdef4827a49e0a71bb21dfb3" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 36217 } }
I20260810 04:06:49.395537 2130 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9117a0f1bdef4827a49e0a71bb21dfb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:06:49.395574 2130 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9117a0f1bdef4827a49e0a71bb21dfb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9117a0f1bdef4827a49e0a71bb21dfb3, State: Initialized, Role: FOLLOWER
I20260810 04:06:49.395725 2130 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9117a0f1bdef4827a49e0a71bb21dfb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9117a0f1bdef4827a49e0a71bb21dfb3" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 36217 } }
I20260810 04:06:49.395833 2130 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9117a0f1bdef4827a49e0a71bb21dfb3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 04:06:49.395867 2130 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9117a0f1bdef4827a49e0a71bb21dfb3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 04:06:49.395918 2130 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9117a0f1bdef4827a49e0a71bb21dfb3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:06:49.396663 2130 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9117a0f1bdef4827a49e0a71bb21dfb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9117a0f1bdef4827a49e0a71bb21dfb3" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 36217 } }
I20260810 04:06:49.396795 2130 leader_election.cc:302] T 00000000000000000000000000000000 P 9117a0f1bdef4827a49e0a71bb21dfb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9117a0f1bdef4827a49e0a71bb21dfb3; no voters:
I20260810 04:06:49.396944 2130 leader_election.cc:288] T 00000000000000000000000000000000 P 9117a0f1bdef4827a49e0a71bb21dfb3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 04:06:49.397014 2136 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9117a0f1bdef4827a49e0a71bb21dfb3 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:06:49.397164 2136 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9117a0f1bdef4827a49e0a71bb21dfb3 [term 1 LEADER]: Becoming Leader. State: Replica: 9117a0f1bdef4827a49e0a71bb21dfb3, State: Running, Role: LEADER
I20260810 04:06:49.397190 2130 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9117a0f1bdef4827a49e0a71bb21dfb3 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:06:49.398701 2136 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9117a0f1bdef4827a49e0a71bb21dfb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9117a0f1bdef4827a49e0a71bb21dfb3" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 36217 } }
I20260810 04:06:49.399317 2138 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9117a0f1bdef4827a49e0a71bb21dfb3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9117a0f1bdef4827a49e0a71bb21dfb3. Latest consensus state: current_term: 1 leader_uuid: "9117a0f1bdef4827a49e0a71bb21dfb3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9117a0f1bdef4827a49e0a71bb21dfb3" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 36217 } } }
I20260810 04:06:49.399308 2139 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9117a0f1bdef4827a49e0a71bb21dfb3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9117a0f1bdef4827a49e0a71bb21dfb3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9117a0f1bdef4827a49e0a71bb21dfb3" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 36217 } } }
I20260810 04:06:49.399554 2139 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9117a0f1bdef4827a49e0a71bb21dfb3 [sys.catalog]: This master's current role is: LEADER
I20260810 04:06:49.399605 2134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:49.388538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47204 (local address 127.31.32.62:36217)
0810 04:06:49.388712 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0810 04:06:49.388721 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 04:06:49.388755 (+ 34us) server_negotiation.cc:407] Connection header received
0810 04:06:49.389212 (+ 457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:06:49.389218 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:06:49.389262 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:06:49.389373 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:06:49.390174 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:49.390982 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:49.392046 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:49.392519 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:49.392642 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:06:49.395090 (+ 2448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:06:49.395094 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:06:49.395094 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:06:49.395106 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:06:49.397330 (+ 2224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:06:49.397782 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:06:49.397785 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:06:49.397785 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:06:49.397916 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:06:49.398149 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:06:49.398150 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:06:49.398150 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:06:49.398504 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:06:49.399018 (+ 514us) server_negotiation.cc:1035] Waiting for connection context
0810 04:06:49.399072 (+ 54us) server_negotiation.cc:299] Negotiation successful
0810 04:06:49.399219 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":90,"threads_started":1}
I20260810 04:06:49.399816 2153 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 04:06:49.399947 2138 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9117a0f1bdef4827a49e0a71bb21dfb3 [sys.catalog]: This master's current role is: LEADER
I20260810 04:06:49.400506 2153 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 04:06:49.401818 2153 catalog_manager.cc:1347] Generated new cluster ID: b8c7dfceaa5e4ebaadda16495c6e3104
I20260810 04:06:49.401885 2153 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 04:06:49.447899 2153 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 04:06:49.448501 2153 catalog_manager.cc:1504] Loading token signing keys...
I20260810 04:06:49.456871 2153 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9117a0f1bdef4827a49e0a71bb21dfb3: Generated new TSK 0
I20260810 04:06:49.457042 2153 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.32.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.32.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.32.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.1.keytab.
Entry for principal kudu/127.31.32.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.1.keytab.
I20260810 04:06:49.483095 31872 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskws89Io/build/release/bin/kudu
/tmp/dist-test-taskws89Io/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786334794676589-31872-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786334794676589-31872-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786334794676589-31872-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786334794676589-31872-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.32.1:0
--local_ip_for_outbound_sockets=127.31.32.1
--webserver_interface=127.31.32.1
--webserver_port=0
--tserver_master_addrs=127.31.32.62:36217
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.1.keytab
--principal=kudu/127.31.32.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.32.20:39159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786334794676589-31872-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786334794676589-31872-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786334794676589-31872-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:06:49.565011 2160 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:06:49.565196 2160 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:06:49.565218 2160 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:06:49.566386 2160 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:06:49.566429 2160 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:06:49.566455 2160 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.32.1
I20260810 04:06:49.567991 2160 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.32.20:39159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786334794676589-31872-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786334794676589-31872-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.1.keytab
--principal=kudu/127.31.32.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.32.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786334794676589-31872-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.32.1
--webserver_port=0
--tserver_master_addrs=127.31.32.62:36217
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2160
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.32.1
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786334794676589-31872-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:06:49.568252 2160 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:06:49.568481 2160 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:06:49.573545 2166 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:06:49.573601 2160 server_base.cc:1034] running on GCE node
W20260810 04:06:49.573567 2168 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:06:49.573585 2165 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:06:49.573925 2160 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:06:49.574182 2160 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:06:49.575353 2160 hybrid_clock.cc:648] HybridClock initialized: now 1786334809575346 us; error 35 us; skew 500 ppm
Aug 10 04:06:49 dist-test-slave-8w9v krb5kdc[2041](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334809, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:06:49.577898 2160 init.cc:377] Logged in from keytab as kudu/127.31.32.1@KRBTEST.COM (short username kudu)
I20260810 04:06:49.578266 2160 webserver.cc:460] Webserver started at http://127.31.32.1:44479/ using document root <none> and password file <none>
I20260810 04:06:49.578464 2160 fs_manager.cc:359] Metadata directory not provided
I20260810 04:06:49.578504 2160 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:06:49.578591 2160 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:06:49.579536 2160 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786334794676589-31872-0/minicluster-data/ts-0/data/instance:
uuid: "a0866b5034554464a7cc6e1be4025eca"
format_stamp: "Formatted at 2026-08-10 04:06:49 on dist-test-slave-8w9v"
server_key: "59b17a49ad33cf8207ab1571b64b4421"
server_key_iv: "bc6e6ce002e569096f4eddb543b691d1"
server_key_version: "encryptionkey@0"
I20260810 04:06:49.579838 2160 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786334794676589-31872-0/minicluster-data/ts-0/wal/instance:
uuid: "a0866b5034554464a7cc6e1be4025eca"
format_stamp: "Formatted at 2026-08-10 04:06:49 on dist-test-slave-8w9v"
server_key: "59b17a49ad33cf8207ab1571b64b4421"
server_key_iv: "bc6e6ce002e569096f4eddb543b691d1"
server_key_version: "encryptionkey@0"
I20260810 04:06:49.581171 2160 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 04:06:49.581821 2175 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:06:49.581971 2160 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 04:06:49.582077 2160 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786334794676589-31872-0/minicluster-data/ts-0/data,/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786334794676589-31872-0/minicluster-data/ts-0/wal
uuid: "a0866b5034554464a7cc6e1be4025eca"
format_stamp: "Formatted at 2026-08-10 04:06:49 on dist-test-slave-8w9v"
server_key: "59b17a49ad33cf8207ab1571b64b4421"
server_key_iv: "bc6e6ce002e569096f4eddb543b691d1"
server_key_version: "encryptionkey@0"
I20260810 04:06:49.582130 2160 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786334794676589-31872-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786334794676589-31872-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786334794676589-31872-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:06:49.592374 2160 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:06:49.594540 2160 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:06:49.594645 2160 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:06:49.594799 2160 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:06:49.595036 2160 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:06:49.595064 2160 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:06:49.595083 2160 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:06:49.595095 2160 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:06:49.601137 2160 rpc_server.cc:307] RPC server started. Bound to: 127.31.32.1:35415
I20260810 04:06:49.601195 2295 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.32.1:35415 every 8 connection(s)
I20260810 04:06:49.601485 2160 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786334794676589-31872-0/minicluster-data/ts-0/data/info.pb
Aug 10 04:06:49 dist-test-slave-8w9v krb5kdc[2041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334809, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.1@KRBTEST.COM for kudu/127.31.32.62@KRBTEST.COM
I20260810 04:06:49.607868 31872 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskws89Io/build/release/bin/kudu as pid 2160
I20260810 04:06:49.607971 31872 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786334794676589-31872-0/minicluster-data/ts-0/wal/instance
I20260810 04:06:49.608170 31872 external_mini_cluster.cc:1428] Setting key 739b50638719e5a82d813f5b9c616e0b
I20260810 04:06:49.610791 2134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:49.602303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.1:46621 (local address 127.31.32.62:36217)
0810 04:06:49.602362 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0810 04:06:49.602364 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:06:49.602527 (+ 163us) server_negotiation.cc:407] Connection header received
0810 04:06:49.602988 (+ 461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:06:49.602989 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:06:49.602998 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:06:49.603048 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:06:49.603890 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:49.604445 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:49.605381 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:49.605567 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:49.605612 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:06:49.607564 (+ 1952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:06:49.607565 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:06:49.607565 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:06:49.607572 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:06:49.609223 (+ 1651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:06:49.609743 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:06:49.609743 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:06:49.609744 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:06:49.609800 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:06:49.610075 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:06:49.610075 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:06:49.610076 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:06:49.610175 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:06:49.610224 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0810 04:06:49.610568 (+ 344us) server_negotiation.cc:299] Negotiation successful
0810 04:06:49.610666 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260810 04:06:49.610940 2298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:49.602235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:36217 (local address 127.31.32.1:46621)
0810 04:06:49.602418 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0810 04:06:49.602458 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0810 04:06:49.602909 (+ 451us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:06:49.603103 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:06:49.603107 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:06:49.603640 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:06:49.603819 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:06:49.603824 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:06:49.604528 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:06:49.604529 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:06:49.605310 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:06:49.605312 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:06:49.605635 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:06:49.605636 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:06:49.605734 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:06:49.606145 (+ 411us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:06:49.606153 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:06:49.607482 (+ 1329us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:06:49.609323 (+ 1841us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:06:49.609330 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:06:49.609348 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:06:49.609664 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:06:49.609867 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:06:49.609868 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:06:49.609869 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:06:49.610008 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:06:49.610237 (+ 229us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:06:49.610243 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:06:49.610409 (+ 166us) client_negotiation.cc:769] Sending connection context
0810 04:06:49.610519 (+ 110us) client_negotiation.cc:240] Negotiation successful
0810 04:06:49.610694 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":105,"threads_started":1}
I20260810 04:06:49.611433 2296 heartbeater.cc:347] Connected to a master server at 127.31.32.62:36217
I20260810 04:06:49.611538 2296 heartbeater.cc:464] Registering TS with master...
I20260810 04:06:49.611729 2296 heartbeater.cc:511] Master 127.31.32.62:36217 requested a full tablet report, sending...
I20260810 04:06:49.612303 2092 ts_manager.cc:194] Registered new tserver with Master: a0866b5034554464a7cc6e1be4025eca (127.31.32.1:35415)
I20260810 04:06:49.612910 2092 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.32.1@KRBTEST.COM'} at 127.31.32.1:46621
WARNING: no policy specified for kudu/127.31.32.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.32.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.32.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.2.keytab.
Entry for principal kudu/127.31.32.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.2.keytab.
I20260810 04:06:49.633198 31872 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskws89Io/build/release/bin/kudu
/tmp/dist-test-taskws89Io/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786334794676589-31872-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786334794676589-31872-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786334794676589-31872-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786334794676589-31872-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.32.2:0
--local_ip_for_outbound_sockets=127.31.32.2
--webserver_interface=127.31.32.2
--webserver_port=0
--tserver_master_addrs=127.31.32.62:36217
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.2.keytab
--principal=kudu/127.31.32.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.32.20:39159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786334794676589-31872-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786334794676589-31872-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786334794676589-31872-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:06:49.711913 2301 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:06:49.712128 2301 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:06:49.712184 2301 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:06:49.713335 2301 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:06:49.713402 2301 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:06:49.713446 2301 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.32.2
I20260810 04:06:49.714875 2301 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.32.20:39159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786334794676589-31872-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786334794676589-31872-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.2.keytab
--principal=kudu/127.31.32.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.32.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786334794676589-31872-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.32.2
--webserver_port=0
--tserver_master_addrs=127.31.32.62:36217
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2301
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.32.2
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786334794676589-31872-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:06:49.715134 2301 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:06:49.715355 2301 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 04:06:49.720289 2301 server_base.cc:1034] running on GCE node
W20260810 04:06:49.720250 2307 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:06:49.720299 2309 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:06:49.720265 2306 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:06:49.720691 2301 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:06:49.720893 2301 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:06:49.722026 2301 hybrid_clock.cc:648] HybridClock initialized: now 1786334809722017 us; error 27 us; skew 500 ppm
Aug 10 04:06:49 dist-test-slave-8w9v krb5kdc[2041](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334809, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:06:49.724530 2301 init.cc:377] Logged in from keytab as kudu/127.31.32.2@KRBTEST.COM (short username kudu)
I20260810 04:06:49.724887 2301 webserver.cc:460] Webserver started at http://127.31.32.2:44073/ using document root <none> and password file <none>
I20260810 04:06:49.725081 2301 fs_manager.cc:359] Metadata directory not provided
I20260810 04:06:49.725121 2301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:06:49.725204 2301 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:06:49.726154 2301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786334794676589-31872-0/minicluster-data/ts-1/data/instance:
uuid: "a9d6a2bb90954d50b68a1ba3ea05532e"
format_stamp: "Formatted at 2026-08-10 04:06:49 on dist-test-slave-8w9v"
server_key: "9666dfd15d9ba630314cb5fefd98395d"
server_key_iv: "8b0adb7741ecec8f24c9eeee0f2217c1"
server_key_version: "encryptionkey@0"
I20260810 04:06:49.726447 2301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786334794676589-31872-0/minicluster-data/ts-1/wal/instance:
uuid: "a9d6a2bb90954d50b68a1ba3ea05532e"
format_stamp: "Formatted at 2026-08-10 04:06:49 on dist-test-slave-8w9v"
server_key: "9666dfd15d9ba630314cb5fefd98395d"
server_key_iv: "8b0adb7741ecec8f24c9eeee0f2217c1"
server_key_version: "encryptionkey@0"
I20260810 04:06:49.727752 2301 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 04:06:49.728459 2317 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:06:49.728634 2301 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 04:06:49.728732 2301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786334794676589-31872-0/minicluster-data/ts-1/data,/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786334794676589-31872-0/minicluster-data/ts-1/wal
uuid: "a9d6a2bb90954d50b68a1ba3ea05532e"
format_stamp: "Formatted at 2026-08-10 04:06:49 on dist-test-slave-8w9v"
server_key: "9666dfd15d9ba630314cb5fefd98395d"
server_key_iv: "8b0adb7741ecec8f24c9eeee0f2217c1"
server_key_version: "encryptionkey@0"
I20260810 04:06:49.728808 2301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786334794676589-31872-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786334794676589-31872-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786334794676589-31872-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:06:49.750716 2301 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:06:49.753404 2301 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:06:49.753571 2301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:06:49.753813 2301 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:06:49.754136 2301 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:06:49.754194 2301 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:06:49.754237 2301 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:06:49.754264 2301 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:06:49.760492 2301 rpc_server.cc:307] RPC server started. Bound to: 127.31.32.2:44181
I20260810 04:06:49.760573 2440 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.32.2:44181 every 8 connection(s)
I20260810 04:06:49.761008 2301 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786334794676589-31872-0/minicluster-data/ts-1/data/info.pb
Aug 10 04:06:49 dist-test-slave-8w9v krb5kdc[2041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334809, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.2@KRBTEST.COM for kudu/127.31.32.62@KRBTEST.COM
I20260810 04:06:49.767685 31872 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskws89Io/build/release/bin/kudu as pid 2301
I20260810 04:06:49.767760 31872 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786334794676589-31872-0/minicluster-data/ts-1/wal/instance
I20260810 04:06:49.767956 31872 external_mini_cluster.cc:1428] Setting key bc4cf5fb77b18c1a1b669fd4d7b21377
I20260810 04:06:49.770562 2134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:49.761743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.2:45287 (local address 127.31.32.62:36217)
0810 04:06:49.761811 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0810 04:06:49.761813 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:06:49.762055 (+ 242us) server_negotiation.cc:407] Connection header received
0810 04:06:49.762581 (+ 526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:06:49.762583 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:06:49.762595 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:06:49.762657 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:06:49.763628 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:49.764177 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:49.765159 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:49.765384 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:49.765475 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:06:49.767574 (+ 2099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:06:49.767575 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:06:49.767576 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:06:49.767584 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:06:49.769178 (+ 1594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:06:49.769580 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:06:49.769582 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:06:49.769582 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:06:49.769640 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:06:49.769903 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:06:49.769904 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:06:49.769904 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:06:49.770026 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:06:49.770100 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0810 04:06:49.770391 (+ 291us) server_negotiation.cc:299] Negotiation successful
0810 04:06:49.770470 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260810 04:06:49.770754 2443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:49.761729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:36217 (local address 127.31.32.2:45287)
0810 04:06:49.761935 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0810 04:06:49.761970 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0810 04:06:49.762475 (+ 505us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:06:49.762732 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:06:49.762735 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:06:49.763315 (+ 580us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:06:49.763504 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:06:49.763509 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:06:49.764258 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:06:49.764259 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:06:49.765060 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:06:49.765063 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:06:49.765483 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:06:49.765485 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:06:49.765606 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:06:49.766094 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:06:49.766102 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:06:49.767462 (+ 1360us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:06:49.769265 (+ 1803us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:06:49.769272 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:06:49.769275 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:06:49.769499 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:06:49.769716 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:06:49.769718 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:06:49.769718 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:06:49.769831 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:06:49.770103 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:06:49.770110 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:06:49.770244 (+ 134us) client_negotiation.cc:769] Sending connection context
0810 04:06:49.770355 (+ 111us) client_negotiation.cc:240] Negotiation successful
0810 04:06:49.770510 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":100,"threads_started":1}
I20260810 04:06:49.771425 2441 heartbeater.cc:347] Connected to a master server at 127.31.32.62:36217
I20260810 04:06:49.771536 2441 heartbeater.cc:464] Registering TS with master...
I20260810 04:06:49.771708 2441 heartbeater.cc:511] Master 127.31.32.62:36217 requested a full tablet report, sending...
I20260810 04:06:49.772176 2092 ts_manager.cc:194] Registered new tserver with Master: a9d6a2bb90954d50b68a1ba3ea05532e (127.31.32.2:44181)
I20260810 04:06:49.772562 2092 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.32.2@KRBTEST.COM'} at 127.31.32.2:45287
WARNING: no policy specified for kudu/127.31.32.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.32.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.32.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.3.keytab.
Entry for principal kudu/127.31.32.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.3.keytab.
I20260810 04:06:49.794358 31872 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskws89Io/build/release/bin/kudu
/tmp/dist-test-taskws89Io/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786334794676589-31872-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786334794676589-31872-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786334794676589-31872-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786334794676589-31872-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.32.3:0
--local_ip_for_outbound_sockets=127.31.32.3
--webserver_interface=127.31.32.3
--webserver_port=0
--tserver_master_addrs=127.31.32.62:36217
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.3.keytab
--principal=kudu/127.31.32.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.32.20:39159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786334794676589-31872-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786334794676589-31872-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786334794676589-31872-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:06:49.876974 2446 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:06:49.877173 2446 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:06:49.877197 2446 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:06:49.878363 2446 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:06:49.878408 2446 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:06:49.878435 2446 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.32.3
I20260810 04:06:49.879956 2446 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.32.20:39159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786334794676589-31872-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786334794676589-31872-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.3.keytab
--principal=kudu/127.31.32.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.32.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786334794676589-31872-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.32.3
--webserver_port=0
--tserver_master_addrs=127.31.32.62:36217
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2446
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.32.3
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786334794676589-31872-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:06:49.880255 2446 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:06:49.880510 2446 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:06:49.885600 2455 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:06:49.885569 2453 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:06:49.885645 2452 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:06:49.885576 2446 server_base.cc:1034] running on GCE node
I20260810 04:06:49.886117 2446 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:06:49.886381 2446 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:06:49.887524 2446 hybrid_clock.cc:648] HybridClock initialized: now 1786334809887511 us; error 32 us; skew 500 ppm
Aug 10 04:06:49 dist-test-slave-8w9v krb5kdc[2041](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334809, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:06:49.890151 2446 init.cc:377] Logged in from keytab as kudu/127.31.32.3@KRBTEST.COM (short username kudu)
I20260810 04:06:49.890627 2446 webserver.cc:460] Webserver started at http://127.31.32.3:33139/ using document root <none> and password file <none>
I20260810 04:06:49.890867 2446 fs_manager.cc:359] Metadata directory not provided
I20260810 04:06:49.890935 2446 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:06:49.891098 2446 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:06:49.892174 2446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786334794676589-31872-0/minicluster-data/ts-2/data/instance:
uuid: "87622e3913c04a17ad6e1e6e1a1d68b9"
format_stamp: "Formatted at 2026-08-10 04:06:49 on dist-test-slave-8w9v"
server_key: "3ae5af88a193ac04090080f09c5b85fe"
server_key_iv: "50a2346af25a1b3e0724943d6e53b5ca"
server_key_version: "encryptionkey@0"
I20260810 04:06:49.892599 2446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786334794676589-31872-0/minicluster-data/ts-2/wal/instance:
uuid: "87622e3913c04a17ad6e1e6e1a1d68b9"
format_stamp: "Formatted at 2026-08-10 04:06:49 on dist-test-slave-8w9v"
server_key: "3ae5af88a193ac04090080f09c5b85fe"
server_key_iv: "50a2346af25a1b3e0724943d6e53b5ca"
server_key_version: "encryptionkey@0"
I20260810 04:06:49.893976 2446 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 04:06:49.894701 2463 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:06:49.894871 2446 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 04:06:49.894976 2446 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786334794676589-31872-0/minicluster-data/ts-2/data,/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786334794676589-31872-0/minicluster-data/ts-2/wal
uuid: "87622e3913c04a17ad6e1e6e1a1d68b9"
format_stamp: "Formatted at 2026-08-10 04:06:49 on dist-test-slave-8w9v"
server_key: "3ae5af88a193ac04090080f09c5b85fe"
server_key_iv: "50a2346af25a1b3e0724943d6e53b5ca"
server_key_version: "encryptionkey@0"
I20260810 04:06:49.895057 2446 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786334794676589-31872-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786334794676589-31872-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786334794676589-31872-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:06:49.910640 2446 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:06:49.912613 2446 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:06:49.912770 2446 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:06:49.912993 2446 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:06:49.913312 2446 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:06:49.913368 2446 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:06:49.913410 2446 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:06:49.913439 2446 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:06:49.919514 2446 rpc_server.cc:307] RPC server started. Bound to: 127.31.32.3:44343
I20260810 04:06:49.919559 2582 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.32.3:44343 every 8 connection(s)
I20260810 04:06:49.919901 2446 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786334794676589-31872-0/minicluster-data/ts-2/data/info.pb
Aug 10 04:06:49 dist-test-slave-8w9v krb5kdc[2041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334809, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.3@KRBTEST.COM for kudu/127.31.32.62@KRBTEST.COM
I20260810 04:06:49.929380 2134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:49.920698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.3:39065 (local address 127.31.32.62:36217)
0810 04:06:49.920782 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0810 04:06:49.920784 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:06:49.920953 (+ 169us) server_negotiation.cc:407] Connection header received
0810 04:06:49.921498 (+ 545us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:06:49.921500 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:06:49.921514 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:06:49.921574 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:06:49.922519 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:49.923047 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:49.924095 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:49.924329 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:49.924424 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:06:49.926606 (+ 2182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:06:49.926608 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:06:49.926608 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:06:49.926616 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:06:49.927900 (+ 1284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:06:49.928428 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:06:49.928429 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:06:49.928430 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:06:49.928491 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:06:49.928750 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:06:49.928751 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:06:49.928752 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:06:49.928865 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:06:49.928942 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0810 04:06:49.929202 (+ 260us) server_negotiation.cc:299] Negotiation successful
0810 04:06:49.929291 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260810 04:06:49.929414 31872 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskws89Io/build/release/bin/kudu as pid 2446
I20260810 04:06:49.929538 31872 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786334794676589-31872-0/minicluster-data/ts-2/wal/instance
I20260810 04:06:49.929613 2585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:49.920648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:36217 (local address 127.31.32.3:39065)
0810 04:06:49.920836 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0810 04:06:49.920871 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0810 04:06:49.921386 (+ 515us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:06:49.921643 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:06:49.921646 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:06:49.922212 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:06:49.922407 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:06:49.922411 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:06:49.923167 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:06:49.923168 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:06:49.923981 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:06:49.923984 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:06:49.924437 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:06:49.924438 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:06:49.924559 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:06:49.925093 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:06:49.925101 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:06:49.926490 (+ 1389us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:06:49.928022 (+ 1532us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:06:49.928030 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:06:49.928032 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:06:49.928338 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:06:49.928571 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:06:49.928572 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:06:49.928573 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:06:49.928683 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:06:49.928934 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:06:49.928941 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:06:49.929074 (+ 133us) client_negotiation.cc:769] Sending connection context
0810 04:06:49.929181 (+ 107us) client_negotiation.cc:240] Negotiation successful
0810 04:06:49.929334 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":83,"threads_started":1}
I20260810 04:06:49.929780 31872 external_mini_cluster.cc:1428] Setting key 10cf85a28bb9862e232aaadab671afd4
I20260810 04:06:49.930166 2583 heartbeater.cc:347] Connected to a master server at 127.31.32.62:36217
I20260810 04:06:49.930269 2583 heartbeater.cc:464] Registering TS with master...
I20260810 04:06:49.930469 2583 heartbeater.cc:511] Master 127.31.32.62:36217 requested a full tablet report, sending...
I20260810 04:06:49.930855 2092 ts_manager.cc:194] Registered new tserver with Master: 87622e3913c04a17ad6e1e6e1a1d68b9 (127.31.32.3:44343)
I20260810 04:06:49.931268 2092 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.32.3@KRBTEST.COM'} at 127.31.32.3:39065
I20260810 04:06:49.941560 31872 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260810 04:06:49.954304 2134 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:06:49.953305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47212 (local address 127.31.32.62:36217)
0810 04:06:49.953350 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0810 04:06:49.953352 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:06:49.953479 (+ 127us) server_negotiation.cc:407] Connection header received
0810 04:06:49.953562 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:06:49.953563 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:06:49.953576 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:06:49.953630 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:06:49.954247 (+ 617us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47212: BlockingRecv error: recv got EOF from 127.0.0.1:47212 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
I20260810 04:06:49.955284 31872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskws89Io/build/release/bin/kudu with pid 2160
I20260810 04:06:49.964104 31872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskws89Io/build/release/bin/kudu with pid 2301
I20260810 04:06:49.969170 31872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskws89Io/build/release/bin/kudu with pid 2446
I20260810 04:06:49.974670 31872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskws89Io/build/release/bin/kudu with pid 2055
2026-08-10T04:06:49Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2903 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786334794676589-31872-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 04:06:50 dist-test-slave-8w9v krb5kdc[2597](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 04:06:50 dist-test-slave-8w9v krb5kdc[2597](info): set up 2 sockets
Aug 10 04:06:50 dist-test-slave-8w9v krb5kdc[2597](info): commencing operation
krb5kdc: starting...
W20260810 04:06:51.053539 31872 mini_kdc.cc:121] Time spent starting KDC: real 1.057s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786334794676589-31872-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786334794676589-31872-0/krb5kdc/test-user.keytab.
Aug 10 04:06:51 dist-test-slave-8w9v krb5kdc[2597](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334811, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T04:06:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T04:06:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.32.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.32.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.32.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.62.keytab.
Entry for principal kudu/127.31.32.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.62.keytab.
I20260810 04:06:51.170797 31872 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskws89Io/build/release/bin/kudu
/tmp/dist-test-taskws89Io/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786334794676589-31872-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786334794676589-31872-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786334794676589-31872-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786334794676589-31872-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.32.62:39387
--webserver_interface=127.31.32.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.62.keytab
--principal=kudu/127.31.32.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.32.20:45439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.32.62:39387
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786334794676589-31872-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786334794676589-31872-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786334794676589-31872-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:06:51.250764 2610 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:06:51.250962 2610 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:06:51.250983 2610 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:06:51.252167 2610 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:06:51.252206 2610 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:06:51.252219 2610 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:06:51.252233 2610 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:06:51.252244 2610 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:06:51.253741 2610 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.32.20:45439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786334794676589-31872-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786334794676589-31872-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.32.62:39387
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.62.keytab
--principal=kudu/127.31.32.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.32.62:39387
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786334794676589-31872-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.32.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2610
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786334794676589-31872-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:06:51.254009 2610 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:06:51.254243 2610 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:06:51.259199 2616 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:06:51.259227 2618 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:06:51.259249 2620 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:06:51.259456 2610 server_base.cc:1034] running on GCE node
I20260810 04:06:51.259696 2610 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:06:51.260008 2610 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:06:51.261176 2610 hybrid_clock.cc:648] HybridClock initialized: now 1786334811261169 us; error 39 us; skew 500 ppm
Aug 10 04:06:51 dist-test-slave-8w9v krb5kdc[2597](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334811, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:06:51.263572 2610 init.cc:377] Logged in from keytab as kudu/127.31.32.62@KRBTEST.COM (short username kudu)
I20260810 04:06:51.263990 2610 webserver.cc:460] Webserver started at http://127.31.32.62:36293/ using document root <none> and password file <none>
I20260810 04:06:51.264274 2610 fs_manager.cc:359] Metadata directory not provided
I20260810 04:06:51.264343 2610 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:06:51.264490 2610 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:06:51.265602 2610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786334794676589-31872-0/minicluster-data/master-0/data/instance:
uuid: "d416cd954eb2488493c2c3564702f7fe"
format_stamp: "Formatted at 2026-08-10 04:06:51 on dist-test-slave-8w9v"
server_key: "0a1e302fafc78513de71eb5dd6770f7d"
server_key_iv: "f404477008e90e9d0b80f9cf11f75fa2"
server_key_version: "encryptionkey@0"
I20260810 04:06:51.265985 2610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786334794676589-31872-0/minicluster-data/master-0/wal/instance:
uuid: "d416cd954eb2488493c2c3564702f7fe"
format_stamp: "Formatted at 2026-08-10 04:06:51 on dist-test-slave-8w9v"
server_key: "0a1e302fafc78513de71eb5dd6770f7d"
server_key_iv: "f404477008e90e9d0b80f9cf11f75fa2"
server_key_version: "encryptionkey@0"
I20260810 04:06:51.267318 2610 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 04:06:51.268013 2627 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:06:51.268216 2610 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 04:06:51.268311 2610 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786334794676589-31872-0/minicluster-data/master-0/data,/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786334794676589-31872-0/minicluster-data/master-0/wal
uuid: "d416cd954eb2488493c2c3564702f7fe"
format_stamp: "Formatted at 2026-08-10 04:06:51 on dist-test-slave-8w9v"
server_key: "0a1e302fafc78513de71eb5dd6770f7d"
server_key_iv: "f404477008e90e9d0b80f9cf11f75fa2"
server_key_version: "encryptionkey@0"
I20260810 04:06:51.268394 2610 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786334794676589-31872-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786334794676589-31872-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786334794676589-31872-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:06:51.279057 2610 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:06:51.281240 2610 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:06:51.281391 2610 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:06:51.285307 2610 rpc_server.cc:307] RPC server started. Bound to: 127.31.32.62:39387
I20260810 04:06:51.285339 2684 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.32.62:39387 every 8 connection(s)
I20260810 04:06:51.285917 2610 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786334794676589-31872-0/minicluster-data/master-0/data/info.pb
I20260810 04:06:51.286594 2685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:06:51.288892 2685 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d416cd954eb2488493c2c3564702f7fe: Bootstrap starting.
I20260810 04:06:51.289477 2685 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d416cd954eb2488493c2c3564702f7fe: Neither blocks nor log segments found. Creating new log.
I20260810 04:06:51.289776 2685 log.cc:824] T 00000000000000000000000000000000 P d416cd954eb2488493c2c3564702f7fe: Log is configured to *not* fsync() on all Append() calls
I20260810 04:06:51.290477 2685 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d416cd954eb2488493c2c3564702f7fe: No bootstrap required, opened a new log
I20260810 04:06:51.292042 2685 raft_consensus.cc:348] T 00000000000000000000000000000000 P d416cd954eb2488493c2c3564702f7fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d416cd954eb2488493c2c3564702f7fe" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 39387 } }
I20260810 04:06:51.292200 2685 raft_consensus.cc:374] T 00000000000000000000000000000000 P d416cd954eb2488493c2c3564702f7fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:06:51.292223 2685 raft_consensus.cc:729] T 00000000000000000000000000000000 P d416cd954eb2488493c2c3564702f7fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d416cd954eb2488493c2c3564702f7fe, State: Initialized, Role: FOLLOWER
I20260810 04:06:51.292407 2685 consensus_queue.cc:260] T 00000000000000000000000000000000 P d416cd954eb2488493c2c3564702f7fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d416cd954eb2488493c2c3564702f7fe" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 39387 } }
I20260810 04:06:51.292470 2685 raft_consensus.cc:388] T 00000000000000000000000000000000 P d416cd954eb2488493c2c3564702f7fe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 04:06:51.292527 2685 raft_consensus.cc:482] T 00000000000000000000000000000000 P d416cd954eb2488493c2c3564702f7fe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 04:06:51.292574 2685 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d416cd954eb2488493c2c3564702f7fe [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:06:51.293465 2685 raft_consensus.cc:504] T 00000000000000000000000000000000 P d416cd954eb2488493c2c3564702f7fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d416cd954eb2488493c2c3564702f7fe" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 39387 } }
I20260810 04:06:51.293581 2685 leader_election.cc:302] T 00000000000000000000000000000000 P d416cd954eb2488493c2c3564702f7fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d416cd954eb2488493c2c3564702f7fe; no voters:
I20260810 04:06:51.293776 2685 leader_election.cc:288] T 00000000000000000000000000000000 P d416cd954eb2488493c2c3564702f7fe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 04:06:51.293823 2688 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d416cd954eb2488493c2c3564702f7fe [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:06:51.293975 2688 raft_consensus.cc:686] T 00000000000000000000000000000000 P d416cd954eb2488493c2c3564702f7fe [term 1 LEADER]: Becoming Leader. State: Replica: d416cd954eb2488493c2c3564702f7fe, State: Running, Role: LEADER
I20260810 04:06:51.294036 2685 sys_catalog.cc:564] T 00000000000000000000000000000000 P d416cd954eb2488493c2c3564702f7fe [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:06:51.294104 2688 consensus_queue.cc:237] T 00000000000000000000000000000000 P d416cd954eb2488493c2c3564702f7fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d416cd954eb2488493c2c3564702f7fe" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 39387 } }
I20260810 04:06:51.294466 2690 sys_catalog.cc:455] T 00000000000000000000000000000000 P d416cd954eb2488493c2c3564702f7fe [sys.catalog]: SysCatalogTable state changed. Reason: New leader d416cd954eb2488493c2c3564702f7fe. Latest consensus state: current_term: 1 leader_uuid: "d416cd954eb2488493c2c3564702f7fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d416cd954eb2488493c2c3564702f7fe" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 39387 } } }
I20260810 04:06:51.294452 2689 sys_catalog.cc:455] T 00000000000000000000000000000000 P d416cd954eb2488493c2c3564702f7fe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d416cd954eb2488493c2c3564702f7fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d416cd954eb2488493c2c3564702f7fe" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 39387 } } }
I20260810 04:06:51.294641 2689 sys_catalog.cc:458] T 00000000000000000000000000000000 P d416cd954eb2488493c2c3564702f7fe [sys.catalog]: This master's current role is: LEADER
I20260810 04:06:51.294987 2697 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 04:06:51.295012 2690 sys_catalog.cc:458] T 00000000000000000000000000000000 P d416cd954eb2488493c2c3564702f7fe [sys.catalog]: This master's current role is: LEADER
I20260810 04:06:51.295478 31872 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskws89Io/build/release/bin/kudu as pid 2610
I20260810 04:06:51.295516 2697 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 04:06:51.295584 31872 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786334794676589-31872-0/minicluster-data/master-0/wal/instance
I20260810 04:06:51.295790 31872 external_mini_cluster.cc:1428] Setting key 20341a0585edaf39f45bc177fc5d2557
I20260810 04:06:51.297843 2697 catalog_manager.cc:1347] Generated new cluster ID: a6c54ad836bf4baa80909bc44e7aa950
I20260810 04:06:51.297895 2697 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 10 04:06:51 dist-test-slave-8w9v krb5kdc[2597](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334811, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.32.62@KRBTEST.COM
I20260810 04:06:51.307417 2707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:51.296265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42954 (local address 127.31.32.62:39387)
0810 04:06:51.296781 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0810 04:06:51.296790 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 04:06:51.296824 (+ 34us) server_negotiation.cc:407] Connection header received
0810 04:06:51.296933 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:06:51.296941 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:06:51.296996 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:06:51.297136 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:06:51.297934 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:51.299291 (+ 1357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:51.300402 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:51.301018 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:51.301116 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:06:51.303972 (+ 2856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:06:51.303977 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:06:51.303977 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:06:51.303989 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:06:51.305857 (+ 1868us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:06:51.306226 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:06:51.306229 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:06:51.306229 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:06:51.306330 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:06:51.306534 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:06:51.306535 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:06:51.306535 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:06:51.306838 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:06:51.306985 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0810 04:06:51.307056 (+ 71us) server_negotiation.cc:299] Negotiation successful
0810 04:06:51.307183 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":450,"thread_start_us":419,"threads_started":1}
I20260810 04:06:51.308295 31872 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 04:06:51.310144 2697 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 04:06:51.310807 2697 catalog_manager.cc:1504] Loading token signing keys...
I20260810 04:06:51.318183 2697 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d416cd954eb2488493c2c3564702f7fe: Generated new TSK 0
I20260810 04:06:51.318414 2697 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 04:06:51.324467 2707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:51.320901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42956 (local address 127.31.32.62:39387)
0810 04:06:51.320956 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0810 04:06:51.320959 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:06:51.321103 (+ 144us) server_negotiation.cc:407] Connection header received
0810 04:06:51.321165 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:06:51.321166 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:06:51.321174 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:06:51.321215 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:06:51.321594 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:51.322060 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:51.323089 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:51.323356 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:51.323410 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:06:51.324012 (+ 602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:06:51.324013 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:06:51.324013 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:06:51.324223 (+ 210us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:06:51.324238 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:06:51.324244 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:06:51.324290 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0810 04:06:51.324382 (+ 92us) server_negotiation.cc:299] Negotiation successful
0810 04:06:51.324401 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260810 04:06:51.326746 31872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskws89Io/build/release/bin/kudu with pid 2610
2026-08-10T04:06:51Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1349 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786334794676589-31872-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 04:06:51 dist-test-slave-8w9v krb5kdc[2718](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 04:06:51 dist-test-slave-8w9v krb5kdc[2718](info): set up 2 sockets
Aug 10 04:06:51 dist-test-slave-8w9v krb5kdc[2718](info): commencing operation
krb5kdc: starting...
W20260810 04:06:52.421186 31872 mini_kdc.cc:121] Time spent starting KDC: real 1.075s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786334794676589-31872-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786334794676589-31872-0/krb5kdc/test-user.keytab.
Aug 10 04:06:52 dist-test-slave-8w9v krb5kdc[2718](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334812, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T04:06:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T04:06:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.32.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.32.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.32.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.62.keytab.
Entry for principal kudu/127.31.32.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.62.keytab.
I20260810 04:06:52.536753 31872 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskws89Io/build/release/bin/kudu
/tmp/dist-test-taskws89Io/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786334794676589-31872-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786334794676589-31872-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786334794676589-31872-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786334794676589-31872-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.32.62:45185
--webserver_interface=127.31.32.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.62.keytab
--principal=kudu/127.31.32.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.32.20:36829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.32.62:45185
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786334794676589-31872-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786334794676589-31872-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786334794676589-31872-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:06:52.618949 2732 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:06:52.619210 2732 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:06:52.619277 2732 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:06:52.620404 2732 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:06:52.620469 2732 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:06:52.620493 2732 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:06:52.620514 2732 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:06:52.620541 2732 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:06:52.621982 2732 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.32.20:36829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786334794676589-31872-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786334794676589-31872-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.32.62:45185
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.62.keytab
--principal=kudu/127.31.32.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.32.62:45185
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786334794676589-31872-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.32.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2732
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786334794676589-31872-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:06:52.622243 2732 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:06:52.622469 2732 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:06:52.627226 2738 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:06:52.627238 2740 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:06:52.627499 2732 server_base.cc:1034] running on GCE node
W20260810 04:06:52.627264 2737 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:06:52.627779 2732 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:06:52.628072 2732 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:06:52.629254 2732 hybrid_clock.cc:648] HybridClock initialized: now 1786334812629240 us; error 26 us; skew 500 ppm
Aug 10 04:06:52 dist-test-slave-8w9v krb5kdc[2718](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334812, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:06:52.631659 2732 init.cc:377] Logged in from keytab as kudu/127.31.32.62@KRBTEST.COM (short username kudu)
I20260810 04:06:52.632030 2732 webserver.cc:460] Webserver started at http://127.31.32.62:38399/ using document root <none> and password file <none>
I20260810 04:06:52.632282 2732 fs_manager.cc:359] Metadata directory not provided
I20260810 04:06:52.632331 2732 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:06:52.632431 2732 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:06:52.633384 2732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786334794676589-31872-0/minicluster-data/master-0/data/instance:
uuid: "6e1be26ac2a644e1be9c6005b87fa02c"
format_stamp: "Formatted at 2026-08-10 04:06:52 on dist-test-slave-8w9v"
server_key: "88c5d95019d95c340257ebb1682cb7d8"
server_key_iv: "b0b1e05355a5f5e06bbb9b9a456b2702"
server_key_version: "encryptionkey@0"
I20260810 04:06:52.633679 2732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786334794676589-31872-0/minicluster-data/master-0/wal/instance:
uuid: "6e1be26ac2a644e1be9c6005b87fa02c"
format_stamp: "Formatted at 2026-08-10 04:06:52 on dist-test-slave-8w9v"
server_key: "88c5d95019d95c340257ebb1682cb7d8"
server_key_iv: "b0b1e05355a5f5e06bbb9b9a456b2702"
server_key_version: "encryptionkey@0"
I20260810 04:06:52.634915 2732 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260810 04:06:52.635600 2749 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:06:52.635766 2732 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260810 04:06:52.635879 2732 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786334794676589-31872-0/minicluster-data/master-0/data,/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786334794676589-31872-0/minicluster-data/master-0/wal
uuid: "6e1be26ac2a644e1be9c6005b87fa02c"
format_stamp: "Formatted at 2026-08-10 04:06:52 on dist-test-slave-8w9v"
server_key: "88c5d95019d95c340257ebb1682cb7d8"
server_key_iv: "b0b1e05355a5f5e06bbb9b9a456b2702"
server_key_version: "encryptionkey@0"
I20260810 04:06:52.635952 2732 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786334794676589-31872-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786334794676589-31872-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786334794676589-31872-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:06:52.682101 2732 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:06:52.684083 2732 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:06:52.684290 2732 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:06:52.688216 2732 rpc_server.cc:307] RPC server started. Bound to: 127.31.32.62:45185
I20260810 04:06:52.688270 2804 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.32.62:45185 every 8 connection(s)
I20260810 04:06:52.688614 2732 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786334794676589-31872-0/minicluster-data/master-0/data/info.pb
I20260810 04:06:52.689716 2805 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:06:52.691560 31872 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskws89Io/build/release/bin/kudu as pid 2732
I20260810 04:06:52.691660 31872 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786334794676589-31872-0/minicluster-data/master-0/wal/instance
I20260810 04:06:52.691833 31872 external_mini_cluster.cc:1428] Setting key a2eff37a33f3761e287dc19b42069df2
I20260810 04:06:52.692569 2805 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e1be26ac2a644e1be9c6005b87fa02c: Bootstrap starting.
I20260810 04:06:52.693166 2805 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6e1be26ac2a644e1be9c6005b87fa02c: Neither blocks nor log segments found. Creating new log.
I20260810 04:06:52.693507 2805 log.cc:824] T 00000000000000000000000000000000 P 6e1be26ac2a644e1be9c6005b87fa02c: Log is configured to *not* fsync() on all Append() calls
I20260810 04:06:52.694278 2805 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e1be26ac2a644e1be9c6005b87fa02c: No bootstrap required, opened a new log
I20260810 04:06:52.696462 2805 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6e1be26ac2a644e1be9c6005b87fa02c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e1be26ac2a644e1be9c6005b87fa02c" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 45185 } }
I20260810 04:06:52.696594 2805 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6e1be26ac2a644e1be9c6005b87fa02c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:06:52.696615 2805 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6e1be26ac2a644e1be9c6005b87fa02c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e1be26ac2a644e1be9c6005b87fa02c, State: Initialized, Role: FOLLOWER
I20260810 04:06:52.696792 2805 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6e1be26ac2a644e1be9c6005b87fa02c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e1be26ac2a644e1be9c6005b87fa02c" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 45185 } }
I20260810 04:06:52.696903 2805 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6e1be26ac2a644e1be9c6005b87fa02c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 04:06:52.696966 2805 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6e1be26ac2a644e1be9c6005b87fa02c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 04:06:52.697055 2805 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6e1be26ac2a644e1be9c6005b87fa02c [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:06:52.697570 2805 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6e1be26ac2a644e1be9c6005b87fa02c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e1be26ac2a644e1be9c6005b87fa02c" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 45185 } }
I20260810 04:06:52.697680 2805 leader_election.cc:302] T 00000000000000000000000000000000 P 6e1be26ac2a644e1be9c6005b87fa02c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6e1be26ac2a644e1be9c6005b87fa02c; no voters:
I20260810 04:06:52.697891 2805 leader_election.cc:288] T 00000000000000000000000000000000 P 6e1be26ac2a644e1be9c6005b87fa02c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 04:06:52.697912 2810 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6e1be26ac2a644e1be9c6005b87fa02c [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:06:52.698024 2810 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6e1be26ac2a644e1be9c6005b87fa02c [term 1 LEADER]: Becoming Leader. State: Replica: 6e1be26ac2a644e1be9c6005b87fa02c, State: Running, Role: LEADER
I20260810 04:06:52.698158 2805 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6e1be26ac2a644e1be9c6005b87fa02c [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:06:52.698164 2810 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6e1be26ac2a644e1be9c6005b87fa02c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e1be26ac2a644e1be9c6005b87fa02c" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 45185 } }
I20260810 04:06:52.698599 2811 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e1be26ac2a644e1be9c6005b87fa02c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6e1be26ac2a644e1be9c6005b87fa02c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e1be26ac2a644e1be9c6005b87fa02c" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 45185 } } }
I20260810 04:06:52.698843 2811 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e1be26ac2a644e1be9c6005b87fa02c [sys.catalog]: This master's current role is: LEADER
Aug 10 04:06:52 dist-test-slave-8w9v krb5kdc[2718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334812, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.32.62@KRBTEST.COM
I20260810 04:06:52.698760 2812 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e1be26ac2a644e1be9c6005b87fa02c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6e1be26ac2a644e1be9c6005b87fa02c. Latest consensus state: current_term: 1 leader_uuid: "6e1be26ac2a644e1be9c6005b87fa02c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e1be26ac2a644e1be9c6005b87fa02c" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 45185 } } }
I20260810 04:06:52.699327 2812 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e1be26ac2a644e1be9c6005b87fa02c [sys.catalog]: This master's current role is: LEADER
W20260810 04:06:52.700109 2828 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6e1be26ac2a644e1be9c6005b87fa02c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 04:06:52.700230 2828 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 04:06:52.700302 2820 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 04:06:52.700510 2820 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 04:06:52.702106 2820 catalog_manager.cc:1347] Generated new cluster ID: 101ed39107e84e3ca390f010b85913b8
I20260810 04:06:52.702152 2820 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 04:06:52.703321 2808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:52.692359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37898 (local address 127.31.32.62:45185)
0810 04:06:52.692554 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0810 04:06:52.692564 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 04:06:52.692601 (+ 37us) server_negotiation.cc:407] Connection header received
0810 04:06:52.693079 (+ 478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:06:52.693086 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:06:52.693135 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:06:52.693294 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:06:52.694119 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:52.695070 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:52.696252 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:52.696718 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:52.696861 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:06:52.699694 (+ 2833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:06:52.699698 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:06:52.699699 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:06:52.699710 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:06:52.701771 (+ 2061us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:06:52.702201 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:06:52.702203 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:06:52.702204 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:06:52.702304 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:06:52.702512 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:06:52.702513 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:06:52.702513 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:06:52.702742 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:06:52.702911 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0810 04:06:52.703016 (+ 105us) server_negotiation.cc:299] Negotiation successful
0810 04:06:52.703151 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":87,"threads_started":1}
I20260810 04:06:52.710071 2820 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 04:06:52.710728 2820 catalog_manager.cc:1504] Loading token signing keys...
I20260810 04:06:52.716522 2820 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6e1be26ac2a644e1be9c6005b87fa02c: Generated new TSK 0
I20260810 04:06:52.716683 2820 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.32.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.32.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.32.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.1.keytab.
Entry for principal kudu/127.31.32.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.1.keytab.
I20260810 04:06:52.743010 31872 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskws89Io/build/release/bin/kudu
/tmp/dist-test-taskws89Io/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786334794676589-31872-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786334794676589-31872-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786334794676589-31872-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786334794676589-31872-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.32.1:0
--local_ip_for_outbound_sockets=127.31.32.1
--webserver_interface=127.31.32.1
--webserver_port=0
--tserver_master_addrs=127.31.32.62:45185
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.1.keytab
--principal=kudu/127.31.32.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.32.20:36829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786334794676589-31872-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786334794676589-31872-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786334794676589-31872-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:06:52.825882 2833 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:06:52.826092 2833 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:06:52.826115 2833 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:06:52.827447 2833 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:06:52.827493 2833 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:06:52.827522 2833 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.32.1
I20260810 04:06:52.828972 2833 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.32.20:36829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786334794676589-31872-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786334794676589-31872-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.1.keytab
--principal=kudu/127.31.32.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.32.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786334794676589-31872-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.32.1
--webserver_port=0
--tserver_master_addrs=127.31.32.62:45185
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2833
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.32.1
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786334794676589-31872-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:06:52.829182 2833 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:06:52.829380 2833 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:06:52.834232 2838 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:06:52.834232 2841 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:06:52.834481 2833 server_base.cc:1034] running on GCE node
W20260810 04:06:52.834332 2839 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:06:52.834775 2833 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:06:52.835016 2833 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:06:52.836192 2833 hybrid_clock.cc:648] HybridClock initialized: now 1786334812836178 us; error 26 us; skew 500 ppm
Aug 10 04:06:52 dist-test-slave-8w9v krb5kdc[2718](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334812, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:06:52.838651 2833 init.cc:377] Logged in from keytab as kudu/127.31.32.1@KRBTEST.COM (short username kudu)
I20260810 04:06:52.839001 2833 webserver.cc:460] Webserver started at http://127.31.32.1:46869/ using document root <none> and password file <none>
I20260810 04:06:52.839248 2833 fs_manager.cc:359] Metadata directory not provided
I20260810 04:06:52.839294 2833 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:06:52.839389 2833 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:06:52.840381 2833 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786334794676589-31872-0/minicluster-data/ts-0/data/instance:
uuid: "26420fd1be8a4fde9c1e22572a4808f3"
format_stamp: "Formatted at 2026-08-10 04:06:52 on dist-test-slave-8w9v"
server_key: "e04f1cd37d0cc6328e0d55db6c3b023f"
server_key_iv: "edc79a177b166bb489a96228ba8b43c0"
server_key_version: "encryptionkey@0"
I20260810 04:06:52.840682 2833 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786334794676589-31872-0/minicluster-data/ts-0/wal/instance:
uuid: "26420fd1be8a4fde9c1e22572a4808f3"
format_stamp: "Formatted at 2026-08-10 04:06:52 on dist-test-slave-8w9v"
server_key: "e04f1cd37d0cc6328e0d55db6c3b023f"
server_key_iv: "edc79a177b166bb489a96228ba8b43c0"
server_key_version: "encryptionkey@0"
I20260810 04:06:52.841873 2833 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:06:52.842615 2848 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:06:52.842808 2833 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:06:52.842893 2833 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786334794676589-31872-0/minicluster-data/ts-0/data,/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786334794676589-31872-0/minicluster-data/ts-0/wal
uuid: "26420fd1be8a4fde9c1e22572a4808f3"
format_stamp: "Formatted at 2026-08-10 04:06:52 on dist-test-slave-8w9v"
server_key: "e04f1cd37d0cc6328e0d55db6c3b023f"
server_key_iv: "edc79a177b166bb489a96228ba8b43c0"
server_key_version: "encryptionkey@0"
I20260810 04:06:52.842947 2833 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786334794676589-31872-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786334794676589-31872-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786334794676589-31872-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:06:52.862659 2833 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:06:52.864790 2833 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:06:52.864897 2833 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:06:52.865048 2833 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:06:52.865340 2833 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:06:52.865373 2833 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:06:52.865393 2833 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:06:52.865439 2833 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:06:52.870857 2833 rpc_server.cc:307] RPC server started. Bound to: 127.31.32.1:40243
I20260810 04:06:52.870936 2968 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.32.1:40243 every 8 connection(s)
I20260810 04:06:52.871213 2833 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786334794676589-31872-0/minicluster-data/ts-0/data/info.pb
Aug 10 04:06:52 dist-test-slave-8w9v krb5kdc[2718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334812, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.1@KRBTEST.COM for kudu/127.31.32.62@KRBTEST.COM
I20260810 04:06:52.877945 31872 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskws89Io/build/release/bin/kudu as pid 2833
I20260810 04:06:52.878067 31872 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786334794676589-31872-0/minicluster-data/ts-0/wal/instance
I20260810 04:06:52.878271 31872 external_mini_cluster.cc:1428] Setting key ca6536f95726ec18a4277ff146112815
I20260810 04:06:52.880234 2808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:52.871960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.1:47725 (local address 127.31.32.62:45185)
0810 04:06:52.872026 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0810 04:06:52.872028 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:06:52.872220 (+ 192us) server_negotiation.cc:407] Connection header received
0810 04:06:52.872671 (+ 451us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:06:52.872672 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:06:52.872681 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:06:52.872734 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:06:52.873589 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:52.874090 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:52.874980 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:52.875157 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:52.875206 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:06:52.877232 (+ 2026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:06:52.877233 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:06:52.877234 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:06:52.877240 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:06:52.878888 (+ 1648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:06:52.879265 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:06:52.879266 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:06:52.879266 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:06:52.879325 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:06:52.879548 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:06:52.879549 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:06:52.879549 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:06:52.879689 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:06:52.879757 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0810 04:06:52.879972 (+ 215us) server_negotiation.cc:299] Negotiation successful
0810 04:06:52.880064 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260810 04:06:52.880252 2971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:52.871935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:45185 (local address 127.31.32.1:47725)
0810 04:06:52.872104 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0810 04:06:52.872155 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0810 04:06:52.872591 (+ 436us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:06:52.872783 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:06:52.872785 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:06:52.873341 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:06:52.873516 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:06:52.873520 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:06:52.874162 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:06:52.874163 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:06:52.874910 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:06:52.874912 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:06:52.875225 (+ 313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:06:52.875226 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:06:52.875328 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:06:52.875764 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:06:52.875771 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:06:52.877148 (+ 1377us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:06:52.878958 (+ 1810us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:06:52.878965 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:06:52.878966 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:06:52.879193 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:06:52.879398 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:06:52.879398 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:06:52.879398 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:06:52.879487 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:06:52.879751 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:06:52.879755 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:06:52.879856 (+ 101us) client_negotiation.cc:769] Sending connection context
0810 04:06:52.879931 (+ 75us) client_negotiation.cc:240] Negotiation successful
0810 04:06:52.880037 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":93,"threads_started":1}
I20260810 04:06:52.880998 2969 heartbeater.cc:347] Connected to a master server at 127.31.32.62:45185
I20260810 04:06:52.881075 2969 heartbeater.cc:464] Registering TS with master...
I20260810 04:06:52.881265 2969 heartbeater.cc:511] Master 127.31.32.62:45185 requested a full tablet report, sending...
I20260810 04:06:52.881801 2768 ts_manager.cc:194] Registered new tserver with Master: 26420fd1be8a4fde9c1e22572a4808f3 (127.31.32.1:40243)
I20260810 04:06:52.882704 2768 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.32.1@KRBTEST.COM'} at 127.31.32.1:47725
WARNING: no policy specified for kudu/127.31.32.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.32.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.32.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.2.keytab.
Entry for principal kudu/127.31.32.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.2.keytab.
I20260810 04:06:52.903510 31872 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskws89Io/build/release/bin/kudu
/tmp/dist-test-taskws89Io/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786334794676589-31872-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786334794676589-31872-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786334794676589-31872-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786334794676589-31872-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.32.2:0
--local_ip_for_outbound_sockets=127.31.32.2
--webserver_interface=127.31.32.2
--webserver_port=0
--tserver_master_addrs=127.31.32.62:45185
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.2.keytab
--principal=kudu/127.31.32.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.32.20:36829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786334794676589-31872-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786334794676589-31872-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786334794676589-31872-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:06:52.985406 2974 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:06:52.985610 2974 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:06:52.985635 2974 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:06:52.986796 2974 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:06:52.986845 2974 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:06:52.986872 2974 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.32.2
I20260810 04:06:52.988441 2974 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.32.20:36829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786334794676589-31872-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786334794676589-31872-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.2.keytab
--principal=kudu/127.31.32.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.32.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786334794676589-31872-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.32.2
--webserver_port=0
--tserver_master_addrs=127.31.32.62:45185
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2974
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.32.2
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786334794676589-31872-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:06:52.988698 2974 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:06:52.988940 2974 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:06:52.995091 2980 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:06:52.995139 2974 server_base.cc:1034] running on GCE node
W20260810 04:06:52.995213 2981 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:06:52.995402 2983 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:06:52.995622 2974 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:06:52.995868 2974 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:06:52.997044 2974 hybrid_clock.cc:648] HybridClock initialized: now 1786334812997027 us; error 28 us; skew 500 ppm
Aug 10 04:06:52 dist-test-slave-8w9v krb5kdc[2718](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334812, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:06:52.999639 2974 init.cc:377] Logged in from keytab as kudu/127.31.32.2@KRBTEST.COM (short username kudu)
I20260810 04:06:53.000109 2974 webserver.cc:460] Webserver started at http://127.31.32.2:38517/ using document root <none> and password file <none>
I20260810 04:06:53.000370 2974 fs_manager.cc:359] Metadata directory not provided
I20260810 04:06:53.000414 2974 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:06:53.000502 2974 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:06:53.001448 2974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786334794676589-31872-0/minicluster-data/ts-1/data/instance:
uuid: "af279edae87e43389517f0ed98b798b5"
format_stamp: "Formatted at 2026-08-10 04:06:52 on dist-test-slave-8w9v"
server_key: "0760312605c4e690ba1cd752eb10d4ec"
server_key_iv: "b99b21d3f3aafe678d1baee62d2fa705"
server_key_version: "encryptionkey@0"
I20260810 04:06:53.001737 2974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786334794676589-31872-0/minicluster-data/ts-1/wal/instance:
uuid: "af279edae87e43389517f0ed98b798b5"
format_stamp: "Formatted at 2026-08-10 04:06:52 on dist-test-slave-8w9v"
server_key: "0760312605c4e690ba1cd752eb10d4ec"
server_key_iv: "b99b21d3f3aafe678d1baee62d2fa705"
server_key_version: "encryptionkey@0"
I20260810 04:06:53.003006 2974 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 04:06:53.003659 2991 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:06:53.003808 2974 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 04:06:53.003901 2974 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786334794676589-31872-0/minicluster-data/ts-1/data,/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786334794676589-31872-0/minicluster-data/ts-1/wal
uuid: "af279edae87e43389517f0ed98b798b5"
format_stamp: "Formatted at 2026-08-10 04:06:52 on dist-test-slave-8w9v"
server_key: "0760312605c4e690ba1cd752eb10d4ec"
server_key_iv: "b99b21d3f3aafe678d1baee62d2fa705"
server_key_version: "encryptionkey@0"
I20260810 04:06:53.003970 2974 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786334794676589-31872-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786334794676589-31872-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786334794676589-31872-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:06:53.031625 2974 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:06:53.033928 2974 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:06:53.034065 2974 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:06:53.034243 2974 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:06:53.034574 2974 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:06:53.034610 2974 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:06:53.034655 2974 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:06:53.034682 2974 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:06:53.040784 2974 rpc_server.cc:307] RPC server started. Bound to: 127.31.32.2:36405
I20260810 04:06:53.040835 3114 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.32.2:36405 every 8 connection(s)
I20260810 04:06:53.041155 2974 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786334794676589-31872-0/minicluster-data/ts-1/data/info.pb
Aug 10 04:06:53 dist-test-slave-8w9v krb5kdc[2718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334812, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.2@KRBTEST.COM for kudu/127.31.32.62@KRBTEST.COM
I20260810 04:06:53.048389 31872 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskws89Io/build/release/bin/kudu as pid 2974
I20260810 04:06:53.048506 31872 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786334794676589-31872-0/minicluster-data/ts-1/wal/instance
I20260810 04:06:53.048758 31872 external_mini_cluster.cc:1428] Setting key 2d4a1b0c2feeccba9036fd78c13afec6
I20260810 04:06:53.050776 2808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:53.041859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.2:33415 (local address 127.31.32.62:45185)
0810 04:06:53.041947 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0810 04:06:53.041950 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:06:53.042124 (+ 174us) server_negotiation.cc:407] Connection header received
0810 04:06:53.042693 (+ 569us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:06:53.042694 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:06:53.042708 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:06:53.042755 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:06:53.043682 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:53.044275 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:53.045267 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:53.045540 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:53.045604 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:06:53.047794 (+ 2190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:06:53.047802 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:06:53.047802 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:06:53.047810 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:06:53.049309 (+ 1499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:06:53.049815 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:06:53.049816 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:06:53.049817 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:06:53.049882 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:06:53.050174 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:06:53.050174 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:06:53.050175 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:06:53.050272 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:06:53.050324 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0810 04:06:53.050644 (+ 320us) server_negotiation.cc:299] Negotiation successful
0810 04:06:53.050710 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260810 04:06:53.051076 3117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:53.041821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:45185 (local address 127.31.32.2:33415)
0810 04:06:53.042006 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0810 04:06:53.042042 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0810 04:06:53.042575 (+ 533us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:06:53.042831 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:06:53.042834 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:06:53.043396 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:06:53.043581 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:06:53.043587 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:06:53.044375 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:06:53.044378 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:06:53.045176 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:06:53.045178 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:06:53.045629 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:06:53.045630 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:06:53.045752 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:06:53.046177 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:06:53.046186 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:06:53.047704 (+ 1518us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:06:53.049403 (+ 1699us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:06:53.049411 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:06:53.049413 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:06:53.049737 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:06:53.049952 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:06:53.049953 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:06:53.049953 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:06:53.050106 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:06:53.050341 (+ 235us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:06:53.050347 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:06:53.050507 (+ 160us) client_negotiation.cc:769] Sending connection context
0810 04:06:53.050623 (+ 116us) client_negotiation.cc:240] Negotiation successful
0810 04:06:53.050801 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":87,"threads_started":1}
I20260810 04:06:53.051636 3115 heartbeater.cc:347] Connected to a master server at 127.31.32.62:45185
I20260810 04:06:53.051738 3115 heartbeater.cc:464] Registering TS with master...
I20260810 04:06:53.051921 3115 heartbeater.cc:511] Master 127.31.32.62:45185 requested a full tablet report, sending...
I20260810 04:06:53.052338 2768 ts_manager.cc:194] Registered new tserver with Master: af279edae87e43389517f0ed98b798b5 (127.31.32.2:36405)
I20260810 04:06:53.052858 2768 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.32.2@KRBTEST.COM'} at 127.31.32.2:33415
WARNING: no policy specified for kudu/127.31.32.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.32.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.32.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.3.keytab.
Entry for principal kudu/127.31.32.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.3.keytab.
I20260810 04:06:53.075300 31872 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskws89Io/build/release/bin/kudu
/tmp/dist-test-taskws89Io/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786334794676589-31872-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786334794676589-31872-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786334794676589-31872-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786334794676589-31872-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.32.3:0
--local_ip_for_outbound_sockets=127.31.32.3
--webserver_interface=127.31.32.3
--webserver_port=0
--tserver_master_addrs=127.31.32.62:45185
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.3.keytab
--principal=kudu/127.31.32.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.32.20:36829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786334794676589-31872-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786334794676589-31872-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786334794676589-31872-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:06:53.157799 3120 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:06:53.157989 3120 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:06:53.158011 3120 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:06:53.159142 3120 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:06:53.159183 3120 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:06:53.159209 3120 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.32.3
I20260810 04:06:53.160707 3120 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.32.20:36829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786334794676589-31872-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786334794676589-31872-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.3.keytab
--principal=kudu/127.31.32.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.32.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786334794676589-31872-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.32.3
--webserver_port=0
--tserver_master_addrs=127.31.32.62:45185
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3120
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.32.3
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786334794676589-31872-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:06:53.160938 3120 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:06:53.161154 3120 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:06:53.166097 3126 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:06:53.166133 3127 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:06:53.166152 3129 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:06:53.166183 3120 server_base.cc:1034] running on GCE node
I20260810 04:06:53.166559 3120 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:06:53.166788 3120 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:06:53.167963 3120 hybrid_clock.cc:648] HybridClock initialized: now 1786334813167949 us; error 26 us; skew 500 ppm
Aug 10 04:06:53 dist-test-slave-8w9v krb5kdc[2718](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334813, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:06:53.170568 3120 init.cc:377] Logged in from keytab as kudu/127.31.32.3@KRBTEST.COM (short username kudu)
I20260810 04:06:53.171007 3120 webserver.cc:460] Webserver started at http://127.31.32.3:34129/ using document root <none> and password file <none>
I20260810 04:06:53.171241 3120 fs_manager.cc:359] Metadata directory not provided
I20260810 04:06:53.171316 3120 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:06:53.171474 3120 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:06:53.172576 3120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786334794676589-31872-0/minicluster-data/ts-2/data/instance:
uuid: "d43bf3c7fb4f4832acde89730dc83151"
format_stamp: "Formatted at 2026-08-10 04:06:53 on dist-test-slave-8w9v"
server_key: "b8a29111e5e25448b419a66cbb1a5793"
server_key_iv: "1997969265daff42112e6576ff656e5f"
server_key_version: "encryptionkey@0"
I20260810 04:06:53.172938 3120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786334794676589-31872-0/minicluster-data/ts-2/wal/instance:
uuid: "d43bf3c7fb4f4832acde89730dc83151"
format_stamp: "Formatted at 2026-08-10 04:06:53 on dist-test-slave-8w9v"
server_key: "b8a29111e5e25448b419a66cbb1a5793"
server_key_iv: "1997969265daff42112e6576ff656e5f"
server_key_version: "encryptionkey@0"
I20260810 04:06:53.174300 3120 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:06:53.175040 3136 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:06:53.175273 3120 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:06:53.175387 3120 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786334794676589-31872-0/minicluster-data/ts-2/data,/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786334794676589-31872-0/minicluster-data/ts-2/wal
uuid: "d43bf3c7fb4f4832acde89730dc83151"
format_stamp: "Formatted at 2026-08-10 04:06:53 on dist-test-slave-8w9v"
server_key: "b8a29111e5e25448b419a66cbb1a5793"
server_key_iv: "1997969265daff42112e6576ff656e5f"
server_key_version: "encryptionkey@0"
I20260810 04:06:53.175478 3120 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786334794676589-31872-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786334794676589-31872-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786334794676589-31872-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:06:53.189086 3120 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:06:53.191422 3120 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:06:53.191592 3120 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:06:53.191831 3120 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:06:53.192281 3120 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:06:53.192344 3120 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:06:53.192394 3120 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:06:53.192427 3120 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:06:53.199033 3120 rpc_server.cc:307] RPC server started. Bound to: 127.31.32.3:42031
I20260810 04:06:53.199064 3257 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.32.3:42031 every 8 connection(s)
I20260810 04:06:53.199481 3120 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786334794676589-31872-0/minicluster-data/ts-2/data/info.pb
I20260810 04:06:53.200245 31872 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskws89Io/build/release/bin/kudu as pid 3120
I20260810 04:06:53.200366 31872 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786334794676589-31872-0/minicluster-data/ts-2/wal/instance
I20260810 04:06:53.200562 31872 external_mini_cluster.cc:1428] Setting key 9288bb3bcfc87e629e338c4691307db9
Aug 10 04:06:53 dist-test-slave-8w9v krb5kdc[2718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334813, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.3@KRBTEST.COM for kudu/127.31.32.62@KRBTEST.COM
I20260810 04:06:53.208920 2808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:53.200234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.3:45917 (local address 127.31.32.62:45185)
0810 04:06:53.200303 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0810 04:06:53.200305 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:06:53.200564 (+ 259us) server_negotiation.cc:407] Connection header received
0810 04:06:53.201273 (+ 709us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:06:53.201274 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:06:53.201285 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:06:53.201358 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:06:53.202365 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:53.202846 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:53.203972 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:53.204200 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:53.204262 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:06:53.206409 (+ 2147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:06:53.206410 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:06:53.206411 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:06:53.206417 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:06:53.207752 (+ 1335us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:06:53.208110 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:06:53.208111 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:06:53.208111 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:06:53.208212 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:06:53.208416 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:06:53.208417 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:06:53.208417 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:06:53.208515 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:06:53.208565 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0810 04:06:53.208800 (+ 235us) server_negotiation.cc:299] Negotiation successful
0810 04:06:53.208858 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260810 04:06:53.209091 3260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:53.200239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:45185 (local address 127.31.32.3:45917)
0810 04:06:53.200426 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0810 04:06:53.200465 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0810 04:06:53.201145 (+ 680us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:06:53.201446 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:06:53.201451 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:06:53.202083 (+ 632us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:06:53.202283 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:06:53.202288 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:06:53.202927 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:06:53.202928 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:06:53.203895 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:06:53.203898 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:06:53.204288 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:06:53.204289 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:06:53.204393 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:06:53.204931 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:06:53.204939 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:06:53.206322 (+ 1383us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:06:53.207811 (+ 1489us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:06:53.207817 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:06:53.207819 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:06:53.208041 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:06:53.208259 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:06:53.208259 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:06:53.208259 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:06:53.208357 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:06:53.208575 (+ 218us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:06:53.208581 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:06:53.208687 (+ 106us) client_negotiation.cc:769] Sending connection context
0810 04:06:53.208778 (+ 91us) client_negotiation.cc:240] Negotiation successful
0810 04:06:53.208895 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":90,"threads_started":1}
I20260810 04:06:53.209584 3258 heartbeater.cc:347] Connected to a master server at 127.31.32.62:45185
I20260810 04:06:53.209687 3258 heartbeater.cc:464] Registering TS with master...
I20260810 04:06:53.209857 3258 heartbeater.cc:511] Master 127.31.32.62:45185 requested a full tablet report, sending...
I20260810 04:06:53.210223 2768 ts_manager.cc:194] Registered new tserver with Master: d43bf3c7fb4f4832acde89730dc83151 (127.31.32.3:42031)
I20260810 04:06:53.210556 2768 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.32.3@KRBTEST.COM'} at 127.31.32.3:45917
I20260810 04:06:53.212440 31872 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 10 04:06:53 dist-test-slave-8w9v krb5kdc[2718](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334813, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Aug 10 04:06:53 dist-test-slave-8w9v krb5kdc[2718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334813, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.31.32.62@KRBTEST.COM
I20260810 04:06:53.236182 2808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:53.229025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37904 (local address 127.31.32.62:45185)
0810 04:06:53.229129 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0810 04:06:53.229132 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:06:53.229241 (+ 109us) server_negotiation.cc:407] Connection header received
0810 04:06:53.229347 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:06:53.229348 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:06:53.229361 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:06:53.229420 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:06:53.230014 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:53.230502 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:53.231233 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:53.231429 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:53.231505 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:06:53.233760 (+ 2255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:06:53.233761 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:06:53.233762 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:06:53.233769 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:06:53.234953 (+ 1184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:06:53.235312 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:06:53.235313 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:06:53.235314 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:06:53.235370 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:06:53.235575 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:06:53.235575 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:06:53.235576 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:06:53.235724 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:06:53.235829 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0810 04:06:53.235937 (+ 108us) server_negotiation.cc:299] Negotiation successful
0810 04:06:53.236033 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
W20260810 04:06:53.236461 2768 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:37904
I20260810 04:06:53.237061 31872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskws89Io/build/release/bin/kudu with pid 2833
I20260810 04:06:53.243662 31872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskws89Io/build/release/bin/kudu with pid 2974
I20260810 04:06:53.250389 31872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskws89Io/build/release/bin/kudu with pid 3120
I20260810 04:06:53.255208 31872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskws89Io/build/release/bin/kudu with pid 2732
2026-08-10T04:06:53Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1929 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786334794676589-31872-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 04:06:53 dist-test-slave-8w9v krb5kdc[3270](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 04:06:53 dist-test-slave-8w9v krb5kdc[3270](info): set up 2 sockets
Aug 10 04:06:53 dist-test-slave-8w9v krb5kdc[3270](info): commencing operation
krb5kdc: starting...
W20260810 04:06:54.341069 31872 mini_kdc.cc:121] Time spent starting KDC: real 1.065s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786334794676589-31872-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786334794676589-31872-0/krb5kdc/test-user.keytab.
Aug 10 04:06:54 dist-test-slave-8w9v krb5kdc[3270](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334814, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T04:06:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T04:06:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.32.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.32.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.32.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.62.keytab.
Entry for principal kudu/127.31.32.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.62.keytab.
I20260810 04:06:54.464641 31872 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskws89Io/build/release/bin/kudu
/tmp/dist-test-taskws89Io/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786334794676589-31872-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786334794676589-31872-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786334794676589-31872-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786334794676589-31872-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.32.62:43183
--webserver_interface=127.31.32.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.62.keytab
--principal=kudu/127.31.32.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.32.20:36895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.32.62:43183
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786334794676589-31872-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786334794676589-31872-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786334794676589-31872-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:06:54.546751 3284 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:06:54.546967 3284 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:06:54.546989 3284 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:06:54.548046 3284 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:06:54.548081 3284 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:06:54.548095 3284 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:06:54.548107 3284 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:06:54.548169 3284 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:06:54.549764 3284 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.32.20:36895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786334794676589-31872-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786334794676589-31872-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.32.62:43183
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.62.keytab
--principal=kudu/127.31.32.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.32.62:43183
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786334794676589-31872-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.32.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3284
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786334794676589-31872-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:06:54.550019 3284 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:06:54.550261 3284 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:06:54.555496 3293 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:06:54.555486 3291 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:06:54.555545 3290 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:06:54.555589 3284 server_base.cc:1034] running on GCE node
I20260810 04:06:54.555842 3284 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:06:54.556104 3284 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:06:54.557302 3284 hybrid_clock.cc:648] HybridClock initialized: now 1786334814557294 us; error 39 us; skew 500 ppm
Aug 10 04:06:54 dist-test-slave-8w9v krb5kdc[3270](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334814, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:06:54.563299 3284 init.cc:377] Logged in from keytab as kudu/127.31.32.62@KRBTEST.COM (short username kudu)
I20260810 04:06:54.563773 3284 webserver.cc:460] Webserver started at http://127.31.32.62:46623/ using document root <none> and password file <none>
I20260810 04:06:54.564008 3284 fs_manager.cc:359] Metadata directory not provided
I20260810 04:06:54.564074 3284 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:06:54.564247 3284 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:06:54.565272 3284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786334794676589-31872-0/minicluster-data/master-0/data/instance:
uuid: "b18e329b21ec4355b66c9d44c119de1a"
format_stamp: "Formatted at 2026-08-10 04:06:54 on dist-test-slave-8w9v"
server_key: "faf2f9f2a83053b8996b0a61bb4f4e5f"
server_key_iv: "d8dbcec1e8e18b4077ed203479348b2c"
server_key_version: "encryptionkey@0"
I20260810 04:06:54.565642 3284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786334794676589-31872-0/minicluster-data/master-0/wal/instance:
uuid: "b18e329b21ec4355b66c9d44c119de1a"
format_stamp: "Formatted at 2026-08-10 04:06:54 on dist-test-slave-8w9v"
server_key: "faf2f9f2a83053b8996b0a61bb4f4e5f"
server_key_iv: "d8dbcec1e8e18b4077ed203479348b2c"
server_key_version: "encryptionkey@0"
I20260810 04:06:54.567008 3284 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 04:06:54.567756 3301 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:06:54.567934 3284 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 04:06:54.568033 3284 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786334794676589-31872-0/minicluster-data/master-0/data,/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786334794676589-31872-0/minicluster-data/master-0/wal
uuid: "b18e329b21ec4355b66c9d44c119de1a"
format_stamp: "Formatted at 2026-08-10 04:06:54 on dist-test-slave-8w9v"
server_key: "faf2f9f2a83053b8996b0a61bb4f4e5f"
server_key_iv: "d8dbcec1e8e18b4077ed203479348b2c"
server_key_version: "encryptionkey@0"
I20260810 04:06:54.568115 3284 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786334794676589-31872-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786334794676589-31872-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786334794676589-31872-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:06:54.585819 3284 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:06:54.587579 3284 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:06:54.587739 3284 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:06:54.592908 3284 rpc_server.cc:307] RPC server started. Bound to: 127.31.32.62:43183
I20260810 04:06:54.592943 3356 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.32.62:43183 every 8 connection(s)
I20260810 04:06:54.593475 3284 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786334794676589-31872-0/minicluster-data/master-0/data/info.pb
I20260810 04:06:54.594159 3357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:06:54.596524 3357 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b18e329b21ec4355b66c9d44c119de1a: Bootstrap starting.
I20260810 04:06:54.597157 3357 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b18e329b21ec4355b66c9d44c119de1a: Neither blocks nor log segments found. Creating new log.
I20260810 04:06:54.597476 3357 log.cc:824] T 00000000000000000000000000000000 P b18e329b21ec4355b66c9d44c119de1a: Log is configured to *not* fsync() on all Append() calls
I20260810 04:06:54.598188 3357 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b18e329b21ec4355b66c9d44c119de1a: No bootstrap required, opened a new log
I20260810 04:06:54.599471 31872 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskws89Io/build/release/bin/kudu as pid 3284
I20260810 04:06:54.599586 31872 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786334794676589-31872-0/minicluster-data/master-0/wal/instance
I20260810 04:06:54.599776 31872 external_mini_cluster.cc:1428] Setting key d0d8d3d8821a7992b341204b91656475
I20260810 04:06:54.600207 3357 raft_consensus.cc:348] T 00000000000000000000000000000000 P b18e329b21ec4355b66c9d44c119de1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b18e329b21ec4355b66c9d44c119de1a" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 43183 } }
I20260810 04:06:54.600427 3357 raft_consensus.cc:374] T 00000000000000000000000000000000 P b18e329b21ec4355b66c9d44c119de1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:06:54.600524 3357 raft_consensus.cc:729] T 00000000000000000000000000000000 P b18e329b21ec4355b66c9d44c119de1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b18e329b21ec4355b66c9d44c119de1a, State: Initialized, Role: FOLLOWER
I20260810 04:06:54.600699 3357 consensus_queue.cc:260] T 00000000000000000000000000000000 P b18e329b21ec4355b66c9d44c119de1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b18e329b21ec4355b66c9d44c119de1a" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 43183 } }
I20260810 04:06:54.600783 3357 raft_consensus.cc:388] T 00000000000000000000000000000000 P b18e329b21ec4355b66c9d44c119de1a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 04:06:54.600807 3357 raft_consensus.cc:482] T 00000000000000000000000000000000 P b18e329b21ec4355b66c9d44c119de1a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 04:06:54.600878 3357 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b18e329b21ec4355b66c9d44c119de1a [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:06:54.601586 3357 raft_consensus.cc:504] T 00000000000000000000000000000000 P b18e329b21ec4355b66c9d44c119de1a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b18e329b21ec4355b66c9d44c119de1a" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 43183 } }
I20260810 04:06:54.601706 3357 leader_election.cc:302] T 00000000000000000000000000000000 P b18e329b21ec4355b66c9d44c119de1a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b18e329b21ec4355b66c9d44c119de1a; no voters:
I20260810 04:06:54.601883 3357 leader_election.cc:288] T 00000000000000000000000000000000 P b18e329b21ec4355b66c9d44c119de1a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 04:06:54.601950 3362 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b18e329b21ec4355b66c9d44c119de1a [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:06:54.602097 3362 raft_consensus.cc:686] T 00000000000000000000000000000000 P b18e329b21ec4355b66c9d44c119de1a [term 1 LEADER]: Becoming Leader. State: Replica: b18e329b21ec4355b66c9d44c119de1a, State: Running, Role: LEADER
I20260810 04:06:54.602164 3357 sys_catalog.cc:564] T 00000000000000000000000000000000 P b18e329b21ec4355b66c9d44c119de1a [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:06:54.602221 3362 consensus_queue.cc:237] T 00000000000000000000000000000000 P b18e329b21ec4355b66c9d44c119de1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b18e329b21ec4355b66c9d44c119de1a" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 43183 } }
I20260810 04:06:54.602627 3362 sys_catalog.cc:455] T 00000000000000000000000000000000 P b18e329b21ec4355b66c9d44c119de1a [sys.catalog]: SysCatalogTable state changed. Reason: New leader b18e329b21ec4355b66c9d44c119de1a. Latest consensus state: current_term: 1 leader_uuid: "b18e329b21ec4355b66c9d44c119de1a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b18e329b21ec4355b66c9d44c119de1a" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 43183 } } }
I20260810 04:06:54.602730 3362 sys_catalog.cc:458] T 00000000000000000000000000000000 P b18e329b21ec4355b66c9d44c119de1a [sys.catalog]: This master's current role is: LEADER
I20260810 04:06:54.602638 3363 sys_catalog.cc:455] T 00000000000000000000000000000000 P b18e329b21ec4355b66c9d44c119de1a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b18e329b21ec4355b66c9d44c119de1a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b18e329b21ec4355b66c9d44c119de1a" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 43183 } } }
I20260810 04:06:54.602900 3363 sys_catalog.cc:458] T 00000000000000000000000000000000 P b18e329b21ec4355b66c9d44c119de1a [sys.catalog]: This master's current role is: LEADER
I20260810 04:06:54.603025 3371 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 04:06:54.603652 3371 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 04:06:54.605715 3371 catalog_manager.cc:1347] Generated new cluster ID: 207dd097e35142bdb1f1b1bb77b18e75
I20260810 04:06:54.605772 3371 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 10 04:06:54 dist-test-slave-8w9v krb5kdc[3270](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334814, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.32.62@KRBTEST.COM
I20260810 04:06:54.612869 3361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:54.600358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46496 (local address 127.31.32.62:43183)
0810 04:06:54.601017 (+ 659us) server_negotiation.cc:206] Beginning negotiation
0810 04:06:54.601027 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 04:06:54.601060 (+ 33us) server_negotiation.cc:407] Connection header received
0810 04:06:54.601169 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:06:54.601174 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:06:54.601214 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:06:54.601333 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:06:54.602277 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:54.603885 (+ 1608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:54.604955 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:54.605460 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:54.605593 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:06:54.608416 (+ 2823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:06:54.608420 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:06:54.608420 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:06:54.608432 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:06:54.610974 (+ 2542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:06:54.611518 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:06:54.611525 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:06:54.611526 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:06:54.611641 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:06:54.611875 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:06:54.611875 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:06:54.611876 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:06:54.612197 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:06:54.612351 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0810 04:06:54.612480 (+ 129us) server_negotiation.cc:299] Negotiation successful
0810 04:06:54.612617 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":608,"thread_start_us":578,"threads_started":1}
I20260810 04:06:54.621008 3371 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 04:06:54.621579 3371 catalog_manager.cc:1504] Loading token signing keys...
I20260810 04:06:54.629729 3371 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b18e329b21ec4355b66c9d44c119de1a: Generated new TSK 0
I20260810 04:06:54.629980 3371 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.32.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.32.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.32.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.1.keytab.
Entry for principal kudu/127.31.32.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.1.keytab.
I20260810 04:06:54.659448 31872 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskws89Io/build/release/bin/kudu
/tmp/dist-test-taskws89Io/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786334794676589-31872-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786334794676589-31872-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786334794676589-31872-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786334794676589-31872-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.32.1:0
--local_ip_for_outbound_sockets=127.31.32.1
--webserver_interface=127.31.32.1
--webserver_port=0
--tserver_master_addrs=127.31.32.62:43183
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.1.keytab
--principal=kudu/127.31.32.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.32.20:36895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786334794676589-31872-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786334794676589-31872-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786334794676589-31872-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:06:54.743659 3384 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:06:54.743853 3384 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:06:54.743877 3384 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:06:54.745110 3384 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:06:54.745152 3384 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:06:54.745177 3384 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.32.1
I20260810 04:06:54.746605 3384 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.32.20:36895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786334794676589-31872-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786334794676589-31872-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.1.keytab
--principal=kudu/127.31.32.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.32.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786334794676589-31872-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.32.1
--webserver_port=0
--tserver_master_addrs=127.31.32.62:43183
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3384
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.32.1
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786334794676589-31872-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:06:54.746784 3384 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:06:54.746948 3384 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:06:54.751758 3392 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:06:54.751784 3389 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:06:54.751786 3390 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:06:54.751982 3384 server_base.cc:1034] running on GCE node
I20260810 04:06:54.752249 3384 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:06:54.752516 3384 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:06:54.753659 3384 hybrid_clock.cc:648] HybridClock initialized: now 1786334814753647 us; error 29 us; skew 500 ppm
Aug 10 04:06:54 dist-test-slave-8w9v krb5kdc[3270](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334814, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:06:54.756233 3384 init.cc:377] Logged in from keytab as kudu/127.31.32.1@KRBTEST.COM (short username kudu)
I20260810 04:06:54.756717 3384 webserver.cc:460] Webserver started at http://127.31.32.1:38243/ using document root <none> and password file <none>
I20260810 04:06:54.756974 3384 fs_manager.cc:359] Metadata directory not provided
I20260810 04:06:54.757019 3384 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:06:54.757189 3384 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:06:54.758245 3384 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786334794676589-31872-0/minicluster-data/ts-0/data/instance:
uuid: "4f375c3154304ba6b9f380944ee6e470"
format_stamp: "Formatted at 2026-08-10 04:06:54 on dist-test-slave-8w9v"
server_key: "1a424136b64836c56f65762b88869298"
server_key_iv: "5f4dcf75b50231c6f50204cd8ebc3703"
server_key_version: "encryptionkey@0"
I20260810 04:06:54.758623 3384 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786334794676589-31872-0/minicluster-data/ts-0/wal/instance:
uuid: "4f375c3154304ba6b9f380944ee6e470"
format_stamp: "Formatted at 2026-08-10 04:06:54 on dist-test-slave-8w9v"
server_key: "1a424136b64836c56f65762b88869298"
server_key_iv: "5f4dcf75b50231c6f50204cd8ebc3703"
server_key_version: "encryptionkey@0"
I20260810 04:06:54.759972 3384 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 04:06:54.760802 3400 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:06:54.761011 3384 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:06:54.761108 3384 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786334794676589-31872-0/minicluster-data/ts-0/data,/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786334794676589-31872-0/minicluster-data/ts-0/wal
uuid: "4f375c3154304ba6b9f380944ee6e470"
format_stamp: "Formatted at 2026-08-10 04:06:54 on dist-test-slave-8w9v"
server_key: "1a424136b64836c56f65762b88869298"
server_key_iv: "5f4dcf75b50231c6f50204cd8ebc3703"
server_key_version: "encryptionkey@0"
I20260810 04:06:54.761198 3384 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786334794676589-31872-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786334794676589-31872-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786334794676589-31872-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:06:54.773700 3384 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:06:54.775570 3384 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:06:54.775754 3384 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:06:54.776003 3384 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:06:54.776341 3384 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:06:54.776376 3384 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:06:54.776398 3384 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:06:54.776446 3384 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:06:54.782480 3384 rpc_server.cc:307] RPC server started. Bound to: 127.31.32.1:41761
I20260810 04:06:54.782529 3522 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.32.1:41761 every 8 connection(s)
I20260810 04:06:54.782835 3384 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786334794676589-31872-0/minicluster-data/ts-0/data/info.pb
I20260810 04:06:54.784947 31872 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskws89Io/build/release/bin/kudu as pid 3384
I20260810 04:06:54.785064 31872 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786334794676589-31872-0/minicluster-data/ts-0/wal/instance
I20260810 04:06:54.785244 31872 external_mini_cluster.cc:1428] Setting key 30686b1c9c621cef454f5c01a2acb8b2
Aug 10 04:06:54 dist-test-slave-8w9v krb5kdc[3270](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334814, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.1@KRBTEST.COM for kudu/127.31.32.62@KRBTEST.COM
I20260810 04:06:54.793776 3361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:54.783552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.1:60173 (local address 127.31.32.62:43183)
0810 04:06:54.783621 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0810 04:06:54.783624 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:06:54.783822 (+ 198us) server_negotiation.cc:407] Connection header received
0810 04:06:54.784864 (+ 1042us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:06:54.784866 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:06:54.784881 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:06:54.784945 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:06:54.785683 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:54.786283 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:54.787831 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:54.788062 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:54.788097 (+ 35us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:06:54.790651 (+ 2554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:06:54.790653 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:06:54.790653 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:06:54.790662 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:06:54.792389 (+ 1727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:06:54.792782 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:06:54.792783 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:06:54.792784 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:06:54.792847 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:06:54.793094 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:06:54.793094 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:06:54.793095 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:06:54.793241 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:06:54.793320 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0810 04:06:54.793573 (+ 253us) server_negotiation.cc:299] Negotiation successful
0810 04:06:54.793664 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260810 04:06:54.793872 3526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:54.783515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:43183 (local address 127.31.32.1:60173)
0810 04:06:54.783708 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0810 04:06:54.783743 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0810 04:06:54.784749 (+ 1006us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:06:54.785022 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:06:54.785026 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:06:54.785374 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:06:54.785589 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:06:54.785595 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:06:54.786389 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:06:54.786390 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:06:54.787714 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:06:54.787718 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:06:54.788232 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:06:54.788233 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:06:54.788343 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:06:54.788891 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:06:54.788902 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:06:54.790549 (+ 1647us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:06:54.792473 (+ 1924us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:06:54.792480 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:06:54.792482 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:06:54.792703 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:06:54.792923 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:06:54.792924 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:06:54.792924 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:06:54.793025 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:06:54.793316 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:06:54.793322 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:06:54.793436 (+ 114us) client_negotiation.cc:769] Sending connection context
0810 04:06:54.793519 (+ 83us) client_negotiation.cc:240] Negotiation successful
0810 04:06:54.793640 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":97,"threads_started":1}
I20260810 04:06:54.794587 3523 heartbeater.cc:347] Connected to a master server at 127.31.32.62:43183
I20260810 04:06:54.794699 3523 heartbeater.cc:464] Registering TS with master...
I20260810 04:06:54.794915 3523 heartbeater.cc:511] Master 127.31.32.62:43183 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.31.32.2@KRBTEST.COM; defaulting to no policy
I20260810 04:06:54.795511 3319 ts_manager.cc:194] Registered new tserver with Master: 4f375c3154304ba6b9f380944ee6e470 (127.31.32.1:41761)
I20260810 04:06:54.796213 3319 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.32.1@KRBTEST.COM'} at 127.31.32.1:60173
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.32.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.32.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.2.keytab.
Entry for principal kudu/127.31.32.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.2.keytab.
I20260810 04:06:54.813215 31872 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskws89Io/build/release/bin/kudu
/tmp/dist-test-taskws89Io/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786334794676589-31872-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786334794676589-31872-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786334794676589-31872-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786334794676589-31872-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.32.2:0
--local_ip_for_outbound_sockets=127.31.32.2
--webserver_interface=127.31.32.2
--webserver_port=0
--tserver_master_addrs=127.31.32.62:43183
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.2.keytab
--principal=kudu/127.31.32.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.32.20:36895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786334794676589-31872-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786334794676589-31872-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786334794676589-31872-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:06:54.895128 3529 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:06:54.895363 3529 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:06:54.895403 3529 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:06:54.896601 3529 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:06:54.896661 3529 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:06:54.896703 3529 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.32.2
I20260810 04:06:54.898182 3529 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.32.20:36895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786334794676589-31872-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786334794676589-31872-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.2.keytab
--principal=kudu/127.31.32.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.32.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786334794676589-31872-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.32.2
--webserver_port=0
--tserver_master_addrs=127.31.32.62:43183
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3529
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.32.2
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786334794676589-31872-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:06:54.898459 3529 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:06:54.898674 3529 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:06:54.903911 3535 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:06:54.903923 3529 server_base.cc:1034] running on GCE node
W20260810 04:06:54.903895 3538 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:06:54.903895 3534 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:06:54.904387 3529 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:06:54.904592 3529 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:06:54.905732 3529 hybrid_clock.cc:648] HybridClock initialized: now 1786334814905721 us; error 28 us; skew 500 ppm
Aug 10 04:06:54 dist-test-slave-8w9v krb5kdc[3270](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334814, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:06:54.908280 3529 init.cc:377] Logged in from keytab as kudu/127.31.32.2@KRBTEST.COM (short username kudu)
I20260810 04:06:54.908753 3529 webserver.cc:460] Webserver started at http://127.31.32.2:46391/ using document root <none> and password file <none>
I20260810 04:06:54.908965 3529 fs_manager.cc:359] Metadata directory not provided
I20260810 04:06:54.909008 3529 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:06:54.909157 3529 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:06:54.910136 3529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786334794676589-31872-0/minicluster-data/ts-1/data/instance:
uuid: "cf496d11f4374858a06699d060984c58"
format_stamp: "Formatted at 2026-08-10 04:06:54 on dist-test-slave-8w9v"
server_key: "07f16746bb0a00e62a170e7cec9f529c"
server_key_iv: "69e901562a317226de8a29f621ffe428"
server_key_version: "encryptionkey@0"
I20260810 04:06:54.910476 3529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786334794676589-31872-0/minicluster-data/ts-1/wal/instance:
uuid: "cf496d11f4374858a06699d060984c58"
format_stamp: "Formatted at 2026-08-10 04:06:54 on dist-test-slave-8w9v"
server_key: "07f16746bb0a00e62a170e7cec9f529c"
server_key_iv: "69e901562a317226de8a29f621ffe428"
server_key_version: "encryptionkey@0"
I20260810 04:06:54.911729 3529 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 04:06:54.912482 3545 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:06:54.912663 3529 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260810 04:06:54.912750 3529 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786334794676589-31872-0/minicluster-data/ts-1/data,/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786334794676589-31872-0/minicluster-data/ts-1/wal
uuid: "cf496d11f4374858a06699d060984c58"
format_stamp: "Formatted at 2026-08-10 04:06:54 on dist-test-slave-8w9v"
server_key: "07f16746bb0a00e62a170e7cec9f529c"
server_key_iv: "69e901562a317226de8a29f621ffe428"
server_key_version: "encryptionkey@0"
I20260810 04:06:54.912815 3529 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786334794676589-31872-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786334794676589-31872-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786334794676589-31872-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:06:54.921111 3529 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:06:54.922941 3529 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:06:54.923100 3529 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:06:54.923312 3529 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:06:54.923635 3529 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:06:54.923667 3529 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:06:54.923688 3529 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:06:54.923734 3529 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:06:54.929701 3529 rpc_server.cc:307] RPC server started. Bound to: 127.31.32.2:45189
I20260810 04:06:54.929749 3662 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.32.2:45189 every 8 connection(s)
I20260810 04:06:54.930053 3529 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786334794676589-31872-0/minicluster-data/ts-1/data/info.pb
Aug 10 04:06:54 dist-test-slave-8w9v krb5kdc[3270](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334814, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.2@KRBTEST.COM for kudu/127.31.32.62@KRBTEST.COM
I20260810 04:06:54.938148 31872 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskws89Io/build/release/bin/kudu as pid 3529
I20260810 04:06:54.938269 31872 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786334794676589-31872-0/minicluster-data/ts-1/wal/instance
I20260810 04:06:54.938457 31872 external_mini_cluster.cc:1428] Setting key 2ddb4d6c91202acc003d2456c6b578b6
I20260810 04:06:54.939575 3361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:54.930914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.2:54733 (local address 127.31.32.62:43183)
0810 04:06:54.930961 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0810 04:06:54.930962 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 04:06:54.931173 (+ 211us) server_negotiation.cc:407] Connection header received
0810 04:06:54.932251 (+ 1078us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:06:54.932253 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:06:54.932263 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:06:54.932351 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:06:54.932897 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:54.933401 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:54.934354 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:54.934558 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:54.934608 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:06:54.936745 (+ 2137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:06:54.936746 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:06:54.936747 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:06:54.936753 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:06:54.938035 (+ 1282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:06:54.938509 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:06:54.938510 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:06:54.938511 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:06:54.938577 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:06:54.938872 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:06:54.938873 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:06:54.938873 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:06:54.938996 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:06:54.939068 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0810 04:06:54.939380 (+ 312us) server_negotiation.cc:299] Negotiation successful
0810 04:06:54.939487 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260810 04:06:54.939726 3666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:54.930896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:43183 (local address 127.31.32.2:54733)
0810 04:06:54.931050 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0810 04:06:54.931085 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0810 04:06:54.932092 (+ 1007us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:06:54.932389 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:06:54.932392 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:06:54.932631 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:06:54.932814 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:06:54.932818 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:06:54.933481 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:06:54.933482 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:06:54.934276 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:06:54.934278 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:06:54.934629 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:06:54.934630 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:06:54.934729 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:06:54.935228 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:06:54.935237 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:06:54.936662 (+ 1425us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:06:54.938124 (+ 1462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:06:54.938130 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:06:54.938132 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:06:54.938436 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:06:54.938664 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:06:54.938665 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:06:54.938665 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:06:54.938816 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:06:54.939081 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:06:54.939088 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:06:54.939229 (+ 141us) client_negotiation.cc:769] Sending connection context
0810 04:06:54.939336 (+ 107us) client_negotiation.cc:240] Negotiation successful
0810 04:06:54.939506 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":92,"threads_started":1}
I20260810 04:06:54.940186 3663 heartbeater.cc:347] Connected to a master server at 127.31.32.62:43183
I20260810 04:06:54.940301 3663 heartbeater.cc:464] Registering TS with master...
I20260810 04:06:54.940479 3663 heartbeater.cc:511] Master 127.31.32.62:43183 requested a full tablet report, sending...
I20260810 04:06:54.940855 3319 ts_manager.cc:194] Registered new tserver with Master: cf496d11f4374858a06699d060984c58 (127.31.32.2:45189)
I20260810 04:06:54.941267 3319 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.32.2@KRBTEST.COM'} at 127.31.32.2:54733
WARNING: no policy specified for kudu/127.31.32.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.32.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.32.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.3.keytab.
Entry for principal kudu/127.31.32.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.3.keytab.
I20260810 04:06:54.963053 31872 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskws89Io/build/release/bin/kudu
/tmp/dist-test-taskws89Io/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786334794676589-31872-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786334794676589-31872-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786334794676589-31872-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786334794676589-31872-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.32.3:0
--local_ip_for_outbound_sockets=127.31.32.3
--webserver_interface=127.31.32.3
--webserver_port=0
--tserver_master_addrs=127.31.32.62:43183
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.3.keytab
--principal=kudu/127.31.32.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.32.20:36895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786334794676589-31872-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786334794676589-31872-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786334794676589-31872-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:06:55.046603 3669 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:06:55.046834 3669 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:06:55.046880 3669 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:06:55.048063 3669 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:06:55.048130 3669 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:06:55.048200 3669 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.32.3
I20260810 04:06:55.049714 3669 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.32.20:36895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786334794676589-31872-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786334794676589-31872-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.3.keytab
--principal=kudu/127.31.32.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.32.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786334794676589-31872-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.32.3
--webserver_port=0
--tserver_master_addrs=127.31.32.62:43183
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3669
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.32.3
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786334794676589-31872-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:06:55.049983 3669 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:06:55.050212 3669 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:06:55.055013 3675 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:06:55.055043 3678 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:06:55.055043 3676 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:06:55.055143 3669 server_base.cc:1034] running on GCE node
I20260810 04:06:55.055433 3669 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:06:55.055683 3669 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:06:55.056820 3669 hybrid_clock.cc:648] HybridClock initialized: now 1786334815056808 us; error 29 us; skew 500 ppm
Aug 10 04:06:55 dist-test-slave-8w9v krb5kdc[3270](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334815, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:06:55.059254 3669 init.cc:377] Logged in from keytab as kudu/127.31.32.3@KRBTEST.COM (short username kudu)
I20260810 04:06:55.059852 3669 webserver.cc:460] Webserver started at http://127.31.32.3:42115/ using document root <none> and password file <none>
I20260810 04:06:55.060081 3669 fs_manager.cc:359] Metadata directory not provided
I20260810 04:06:55.060187 3669 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:06:55.060348 3669 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:06:55.061354 3669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786334794676589-31872-0/minicluster-data/ts-2/data/instance:
uuid: "3f34d00aaf854893b661ec06ad51033b"
format_stamp: "Formatted at 2026-08-10 04:06:55 on dist-test-slave-8w9v"
server_key: "c0d9214e124e5afc2ff16ae3528021d2"
server_key_iv: "647a1db052ab16a3aed0b2e4b6b20e74"
server_key_version: "encryptionkey@0"
I20260810 04:06:55.061729 3669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786334794676589-31872-0/minicluster-data/ts-2/wal/instance:
uuid: "3f34d00aaf854893b661ec06ad51033b"
format_stamp: "Formatted at 2026-08-10 04:06:55 on dist-test-slave-8w9v"
server_key: "c0d9214e124e5afc2ff16ae3528021d2"
server_key_iv: "647a1db052ab16a3aed0b2e4b6b20e74"
server_key_version: "encryptionkey@0"
I20260810 04:06:55.063230 3669 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 04:06:55.063953 3685 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:06:55.064149 3669 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 04:06:55.064281 3669 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786334794676589-31872-0/minicluster-data/ts-2/data,/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786334794676589-31872-0/minicluster-data/ts-2/wal
uuid: "3f34d00aaf854893b661ec06ad51033b"
format_stamp: "Formatted at 2026-08-10 04:06:55 on dist-test-slave-8w9v"
server_key: "c0d9214e124e5afc2ff16ae3528021d2"
server_key_iv: "647a1db052ab16a3aed0b2e4b6b20e74"
server_key_version: "encryptionkey@0"
I20260810 04:06:55.064376 3669 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786334794676589-31872-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786334794676589-31872-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786334794676589-31872-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:06:55.072536 3669 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:06:55.074342 3669 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:06:55.074503 3669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:06:55.074743 3669 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:06:55.075089 3669 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:06:55.075146 3669 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:06:55.075189 3669 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:06:55.075237 3669 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:06:55.081261 3669 rpc_server.cc:307] RPC server started. Bound to: 127.31.32.3:36431
I20260810 04:06:55.081321 3805 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.32.3:36431 every 8 connection(s)
I20260810 04:06:55.081617 3669 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786334794676589-31872-0/minicluster-data/ts-2/data/info.pb
Aug 10 04:06:55 dist-test-slave-8w9v krb5kdc[3270](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334815, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.3@KRBTEST.COM for kudu/127.31.32.62@KRBTEST.COM
I20260810 04:06:55.088080 31872 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskws89Io/build/release/bin/kudu as pid 3669
I20260810 04:06:55.088214 31872 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786334794676589-31872-0/minicluster-data/ts-2/wal/instance
I20260810 04:06:55.088441 31872 external_mini_cluster.cc:1428] Setting key eaf30b64386470d605db40c978aa0bf8
I20260810 04:06:55.091183 3361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:55.082349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.3:43831 (local address 127.31.32.62:43183)
0810 04:06:55.082412 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0810 04:06:55.082415 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:06:55.082595 (+ 180us) server_negotiation.cc:407] Connection header received
0810 04:06:55.083562 (+ 967us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:06:55.083564 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:06:55.083597 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:06:55.083664 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:06:55.084385 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:55.084897 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:55.085897 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:55.086088 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:55.086194 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:06:55.088244 (+ 2050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:06:55.088246 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:06:55.088246 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:06:55.088254 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:06:55.089886 (+ 1632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:06:55.090252 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:06:55.090253 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:06:55.090254 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:06:55.090309 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:06:55.090566 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:06:55.090567 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:06:55.090568 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:06:55.090680 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:06:55.090752 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0810 04:06:55.091015 (+ 263us) server_negotiation.cc:299] Negotiation successful
0810 04:06:55.091099 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260810 04:06:55.091331 3809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:55.082315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:43183 (local address 127.31.32.3:43831)
0810 04:06:55.082468 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0810 04:06:55.082513 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 04:06:55.083444 (+ 931us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:06:55.083741 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:06:55.083745 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:06:55.084015 (+ 270us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:06:55.084279 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:06:55.084284 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:06:55.084994 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:06:55.084995 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:06:55.085779 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:06:55.085781 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:06:55.086202 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:06:55.086203 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:06:55.086336 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:06:55.086791 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:06:55.086798 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:06:55.088092 (+ 1294us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:06:55.089965 (+ 1873us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:06:55.089972 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:06:55.089974 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:06:55.090179 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:06:55.090396 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:06:55.090397 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:06:55.090397 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:06:55.090503 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:06:55.090744 (+ 241us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:06:55.090750 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:06:55.090884 (+ 134us) client_negotiation.cc:769] Sending connection context
0810 04:06:55.090980 (+ 96us) client_negotiation.cc:240] Negotiation successful
0810 04:06:55.091121 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":66,"threads_started":1}
I20260810 04:06:55.091763 3806 heartbeater.cc:347] Connected to a master server at 127.31.32.62:43183
I20260810 04:06:55.091869 3806 heartbeater.cc:464] Registering TS with master...
I20260810 04:06:55.092015 3806 heartbeater.cc:511] Master 127.31.32.62:43183 requested a full tablet report, sending...
I20260810 04:06:55.092396 3319 ts_manager.cc:194] Registered new tserver with Master: 3f34d00aaf854893b661ec06ad51033b (127.31.32.3:36431)
I20260810 04:06:55.092746 3319 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.32.3@KRBTEST.COM'} at 127.31.32.3:43831
I20260810 04:06:55.100659 31872 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 10 04:06:55 dist-test-slave-8w9v krb5kdc[3270](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334815, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 10 04:06:55 dist-test-slave-8w9v krb5kdc[3270](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334815, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.32.1@KRBTEST.COM
I20260810 04:06:55.123643 3814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:55.115963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40650 (local address 127.31.32.1:41761)
0810 04:06:55.116213 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0810 04:06:55.116218 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 04:06:55.116228 (+ 10us) server_negotiation.cc:407] Connection header received
0810 04:06:55.116252 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:06:55.116254 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:06:55.116267 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:06:55.116378 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:06:55.116911 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:55.117608 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:55.118308 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:55.118622 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:55.118678 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:06:55.120966 (+ 2288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:06:55.120975 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:06:55.120976 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:06:55.120987 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:06:55.122430 (+ 1443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:06:55.122803 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:06:55.122806 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:06:55.122807 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:06:55.122883 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:06:55.123085 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:06:55.123086 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:06:55.123086 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:06:55.123221 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:06:55.123388 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0810 04:06:55.123435 (+ 47us) server_negotiation.cc:299] Negotiation successful
0810 04:06:55.123516 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":214,"thread_start_us":110,"threads_started":1}
Aug 10 04:06:55 dist-test-slave-8w9v krb5kdc[3270](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334815, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.32.62@KRBTEST.COM
I20260810 04:06:55.131568 3361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:55.125149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46498 (local address 127.31.32.62:43183)
0810 04:06:55.125209 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0810 04:06:55.125211 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:06:55.125218 (+ 7us) server_negotiation.cc:407] Connection header received
0810 04:06:55.125238 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:06:55.125239 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:06:55.125246 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:06:55.125304 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:06:55.125762 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:55.126227 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:55.126883 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:55.127077 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:55.127137 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:06:55.129260 (+ 2123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:06:55.129261 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:06:55.129261 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:06:55.129267 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:06:55.130612 (+ 1345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:06:55.130946 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:06:55.130946 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:06:55.130946 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:06:55.130994 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:06:55.131156 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:06:55.131156 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:06:55.131157 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:06:55.131288 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:06:55.131375 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0810 04:06:55.131448 (+ 73us) server_negotiation.cc:299] Negotiation successful
0810 04:06:55.131506 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
W20260810 04:06:55.131918 3319 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46498
I20260810 04:06:55.132328 31872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskws89Io/build/release/bin/kudu with pid 3384
I20260810 04:06:55.143463 31872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskws89Io/build/release/bin/kudu with pid 3529
I20260810 04:06:55.148663 31872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskws89Io/build/release/bin/kudu with pid 3669
I20260810 04:06:55.154549 31872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskws89Io/build/release/bin/kudu with pid 3284
2026-08-10T04:06:55Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1899 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786334794676589-31872-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 04:06:55 dist-test-slave-8w9v krb5kdc[3820](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 04:06:55 dist-test-slave-8w9v krb5kdc[3820](info): set up 2 sockets
Aug 10 04:06:55 dist-test-slave-8w9v krb5kdc[3820](info): commencing operation
krb5kdc: starting...
W20260810 04:06:56.215929 31872 mini_kdc.cc:121] Time spent starting KDC: real 1.040s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786334794676589-31872-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786334794676589-31872-0/krb5kdc/test-user.keytab.
Aug 10 04:06:56 dist-test-slave-8w9v krb5kdc[3820](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334816, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T04:06:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T04:06:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.32.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.32.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.32.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.62.keytab.
Entry for principal kudu/127.31.32.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.62.keytab.
I20260810 04:06:56.339694 31872 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskws89Io/build/release/bin/kudu
/tmp/dist-test-taskws89Io/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786334794676589-31872-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786334794676589-31872-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786334794676589-31872-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786334794676589-31872-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.32.62:43109
--webserver_interface=127.31.32.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.62.keytab
--principal=kudu/127.31.32.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.32.20:42433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.32.62:43109
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786334794676589-31872-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786334794676589-31872-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786334794676589-31872-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:06:56.422196 3835 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:06:56.422397 3835 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:06:56.422420 3835 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:06:56.423589 3835 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:06:56.423635 3835 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:06:56.423651 3835 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:06:56.423702 3835 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:06:56.423743 3835 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:06:56.425393 3835 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.32.20:42433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786334794676589-31872-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786334794676589-31872-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.32.62:43109
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.62.keytab
--principal=kudu/127.31.32.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.32.62:43109
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786334794676589-31872-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.32.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3835
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786334794676589-31872-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:06:56.425663 3835 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:06:56.425910 3835 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:06:56.431025 3841 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:06:56.431013 3842 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:06:56.431034 3835 server_base.cc:1034] running on GCE node
W20260810 04:06:56.431016 3844 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:06:56.431509 3835 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:06:56.431779 3835 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:06:56.432946 3835 hybrid_clock.cc:648] HybridClock initialized: now 1786334816432932 us; error 46 us; skew 500 ppm
Aug 10 04:06:56 dist-test-slave-8w9v krb5kdc[3820](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334816, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:06:56.435413 3835 init.cc:377] Logged in from keytab as kudu/127.31.32.62@KRBTEST.COM (short username kudu)
I20260810 04:06:56.435724 3835 fs_manager.cc:359] Metadata directory not provided
I20260810 04:06:56.435789 3835 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:06:56.435933 3835 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:06:56.437011 3835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786334794676589-31872-0/minicluster-data/master-0/data/instance:
uuid: "6f4ed470c94343d29ce287ac5e7a6108"
format_stamp: "Formatted at 2026-08-10 04:06:56 on dist-test-slave-8w9v"
server_key: "1366f041c345dcc86230a3f05482f80f"
server_key_iv: "67a958a030244b75fd6ebc85850ca484"
server_key_version: "encryptionkey@0"
I20260810 04:06:56.437368 3835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786334794676589-31872-0/minicluster-data/master-0/wal/instance:
uuid: "6f4ed470c94343d29ce287ac5e7a6108"
format_stamp: "Formatted at 2026-08-10 04:06:56 on dist-test-slave-8w9v"
server_key: "1366f041c345dcc86230a3f05482f80f"
server_key_iv: "67a958a030244b75fd6ebc85850ca484"
server_key_version: "encryptionkey@0"
I20260810 04:06:56.438650 3835 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 04:06:56.439373 3851 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:06:56.439534 3835 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 04:06:56.439632 3835 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786334794676589-31872-0/minicluster-data/master-0/data,/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786334794676589-31872-0/minicluster-data/master-0/wal
uuid: "6f4ed470c94343d29ce287ac5e7a6108"
format_stamp: "Formatted at 2026-08-10 04:06:56 on dist-test-slave-8w9v"
server_key: "1366f041c345dcc86230a3f05482f80f"
server_key_iv: "67a958a030244b75fd6ebc85850ca484"
server_key_version: "encryptionkey@0"
I20260810 04:06:56.439711 3835 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786334794676589-31872-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786334794676589-31872-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786334794676589-31872-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:06:56.449944 3835 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:06:56.451699 3835 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:06:56.451853 3835 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:06:56.455718 3835 rpc_server.cc:307] RPC server started. Bound to: 127.31.32.62:43109
I20260810 04:06:56.455747 3909 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.32.62:43109 every 8 connection(s)
I20260810 04:06:56.456022 3835 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786334794676589-31872-0/minicluster-data/master-0/data/info.pb
I20260810 04:06:56.457106 3910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:06:56.459226 3910 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f4ed470c94343d29ce287ac5e7a6108: Bootstrap starting.
I20260810 04:06:56.459790 3910 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6f4ed470c94343d29ce287ac5e7a6108: Neither blocks nor log segments found. Creating new log.
I20260810 04:06:56.460089 3910 log.cc:824] T 00000000000000000000000000000000 P 6f4ed470c94343d29ce287ac5e7a6108: Log is configured to *not* fsync() on all Append() calls
I20260810 04:06:56.460778 3910 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f4ed470c94343d29ce287ac5e7a6108: No bootstrap required, opened a new log
I20260810 04:06:56.462805 3910 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6f4ed470c94343d29ce287ac5e7a6108 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f4ed470c94343d29ce287ac5e7a6108" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 43109 } }
I20260810 04:06:56.462937 3910 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6f4ed470c94343d29ce287ac5e7a6108 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:06:56.462960 3910 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6f4ed470c94343d29ce287ac5e7a6108 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f4ed470c94343d29ce287ac5e7a6108, State: Initialized, Role: FOLLOWER
I20260810 04:06:56.463119 3910 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6f4ed470c94343d29ce287ac5e7a6108 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f4ed470c94343d29ce287ac5e7a6108" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 43109 } }
I20260810 04:06:56.463188 3910 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6f4ed470c94343d29ce287ac5e7a6108 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 04:06:56.463229 3910 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6f4ed470c94343d29ce287ac5e7a6108 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 04:06:56.463277 3910 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6f4ed470c94343d29ce287ac5e7a6108 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:06:56.464062 3910 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6f4ed470c94343d29ce287ac5e7a6108 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f4ed470c94343d29ce287ac5e7a6108" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 43109 } }
I20260810 04:06:56.464212 3910 leader_election.cc:302] T 00000000000000000000000000000000 P 6f4ed470c94343d29ce287ac5e7a6108 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6f4ed470c94343d29ce287ac5e7a6108; no voters:
I20260810 04:06:56.464421 3910 leader_election.cc:288] T 00000000000000000000000000000000 P 6f4ed470c94343d29ce287ac5e7a6108 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 04:06:56.464485 3913 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6f4ed470c94343d29ce287ac5e7a6108 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:06:56.464665 3913 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6f4ed470c94343d29ce287ac5e7a6108 [term 1 LEADER]: Becoming Leader. State: Replica: 6f4ed470c94343d29ce287ac5e7a6108, State: Running, Role: LEADER
I20260810 04:06:56.464691 3910 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6f4ed470c94343d29ce287ac5e7a6108 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:06:56.464865 3913 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6f4ed470c94343d29ce287ac5e7a6108 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f4ed470c94343d29ce287ac5e7a6108" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 43109 } }
I20260810 04:06:56.465066 31872 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskws89Io/build/release/bin/kudu as pid 3835
I20260810 04:06:56.465179 31872 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786334794676589-31872-0/minicluster-data/master-0/wal/instance
I20260810 04:06:56.465315 3916 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f4ed470c94343d29ce287ac5e7a6108 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6f4ed470c94343d29ce287ac5e7a6108. Latest consensus state: current_term: 1 leader_uuid: "6f4ed470c94343d29ce287ac5e7a6108" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f4ed470c94343d29ce287ac5e7a6108" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 43109 } } }
I20260810 04:06:56.465399 31872 external_mini_cluster.cc:1428] Setting key 394cda6be96ff6e2481a89da7ea8d225
I20260810 04:06:56.465391 3916 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f4ed470c94343d29ce287ac5e7a6108 [sys.catalog]: This master's current role is: LEADER
I20260810 04:06:56.465957 3924 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 04:06:56.466207 3914 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f4ed470c94343d29ce287ac5e7a6108 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6f4ed470c94343d29ce287ac5e7a6108" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f4ed470c94343d29ce287ac5e7a6108" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 43109 } } }
I20260810 04:06:56.466355 3914 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f4ed470c94343d29ce287ac5e7a6108 [sys.catalog]: This master's current role is: LEADER
I20260810 04:06:56.466491 3924 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 04:06:56.468161 3924 catalog_manager.cc:1347] Generated new cluster ID: 78c179183d6d42c9be9003fdc09038f8
I20260810 04:06:56.468215 3924 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 10 04:06:56 dist-test-slave-8w9v krb5kdc[3820](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334816, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.32.62@KRBTEST.COM
I20260810 04:06:56.477077 3924 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 04:06:56.477634 3924 catalog_manager.cc:1504] Loading token signing keys...
I20260810 04:06:56.477905 3932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:56.466067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59538 (local address 127.31.32.62:43109)
0810 04:06:56.466313 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0810 04:06:56.466321 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 04:06:56.466619 (+ 298us) server_negotiation.cc:407] Connection header received
0810 04:06:56.466826 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:06:56.466845 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:06:56.466912 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:06:56.467020 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:06:56.467945 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:56.469376 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:56.470482 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:56.471087 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:56.471210 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:06:56.473913 (+ 2703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:06:56.473917 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:06:56.473917 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:06:56.473929 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:06:56.475908 (+ 1979us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:06:56.476309 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:06:56.476313 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:06:56.476322 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:06:56.476428 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:06:56.476666 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:06:56.476667 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:06:56.476667 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:06:56.476935 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:06:56.477484 (+ 549us) server_negotiation.cc:1035] Waiting for connection context
0810 04:06:56.477554 (+ 70us) server_negotiation.cc:299] Negotiation successful
0810 04:06:56.477691 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":77,"threads_started":1}
I20260810 04:06:56.480989 3924 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6f4ed470c94343d29ce287ac5e7a6108: Generated new TSK 0
I20260810 04:06:56.481144 3924 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.32.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.32.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.32.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.1.keytab.
Entry for principal kudu/127.31.32.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.1.keytab.
I20260810 04:06:56.509033 31872 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskws89Io/build/release/bin/kudu
/tmp/dist-test-taskws89Io/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786334794676589-31872-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786334794676589-31872-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786334794676589-31872-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786334794676589-31872-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.32.1:0
--local_ip_for_outbound_sockets=127.31.32.1
--webserver_interface=127.31.32.1
--webserver_port=0
--tserver_master_addrs=127.31.32.62:43109
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.1.keytab
--principal=kudu/127.31.32.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.32.20:42433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786334794676589-31872-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786334794676589-31872-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786334794676589-31872-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:06:56.592458 3938 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:06:56.592660 3938 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:06:56.592684 3938 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:06:56.593886 3938 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:06:56.593928 3938 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:06:56.593995 3938 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.32.1
I20260810 04:06:56.595489 3938 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.32.20:42433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786334794676589-31872-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786334794676589-31872-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.1.keytab
--principal=kudu/127.31.32.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.32.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786334794676589-31872-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.32.1
--webserver_port=0
--tserver_master_addrs=127.31.32.62:43109
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3938
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.32.1
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786334794676589-31872-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:06:56.595749 3938 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:06:56.596005 3938 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 04:06:56.600991 3938 server_base.cc:1034] running on GCE node
W20260810 04:06:56.601045 3944 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:06:56.600965 3943 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:06:56.600965 3946 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:06:56.601408 3938 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:06:56.601613 3938 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:06:56.602774 3938 hybrid_clock.cc:648] HybridClock initialized: now 1786334816602752 us; error 51 us; skew 500 ppm
Aug 10 04:06:56 dist-test-slave-8w9v krb5kdc[3820](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334816, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:06:56.605304 3938 init.cc:377] Logged in from keytab as kudu/127.31.32.1@KRBTEST.COM (short username kudu)
I20260810 04:06:56.605641 3938 fs_manager.cc:359] Metadata directory not provided
I20260810 04:06:56.605712 3938 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:06:56.605880 3938 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:06:56.606894 3938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786334794676589-31872-0/minicluster-data/ts-0/data/instance:
uuid: "5156b5685cc744378665b53c64889c0d"
format_stamp: "Formatted at 2026-08-10 04:06:56 on dist-test-slave-8w9v"
server_key: "18efe7f2b241a0bf6b403e2a808fa372"
server_key_iv: "c95eb182f4892a6a0ac17ec126b4d6ce"
server_key_version: "encryptionkey@0"
I20260810 04:06:56.607249 3938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786334794676589-31872-0/minicluster-data/ts-0/wal/instance:
uuid: "5156b5685cc744378665b53c64889c0d"
format_stamp: "Formatted at 2026-08-10 04:06:56 on dist-test-slave-8w9v"
server_key: "18efe7f2b241a0bf6b403e2a808fa372"
server_key_iv: "c95eb182f4892a6a0ac17ec126b4d6ce"
server_key_version: "encryptionkey@0"
I20260810 04:06:56.608594 3938 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 04:06:56.609278 3952 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:06:56.609448 3938 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 04:06:56.609548 3938 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786334794676589-31872-0/minicluster-data/ts-0/data,/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786334794676589-31872-0/minicluster-data/ts-0/wal
uuid: "5156b5685cc744378665b53c64889c0d"
format_stamp: "Formatted at 2026-08-10 04:06:56 on dist-test-slave-8w9v"
server_key: "18efe7f2b241a0bf6b403e2a808fa372"
server_key_iv: "c95eb182f4892a6a0ac17ec126b4d6ce"
server_key_version: "encryptionkey@0"
I20260810 04:06:56.609620 3938 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786334794676589-31872-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786334794676589-31872-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786334794676589-31872-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:06:56.633458 3938 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:06:56.635754 3938 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:06:56.635926 3938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:06:56.636236 3938 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:06:56.636564 3938 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:06:56.636617 3938 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:06:56.636654 3938 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:06:56.636682 3938 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:06:56.642688 3938 rpc_server.cc:307] RPC server started. Bound to: 127.31.32.1:41111
I20260810 04:06:56.642724 4073 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.32.1:41111 every 8 connection(s)
I20260810 04:06:56.643051 3938 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786334794676589-31872-0/minicluster-data/ts-0/data/info.pb
I20260810 04:06:56.643713 31872 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskws89Io/build/release/bin/kudu as pid 3938
I20260810 04:06:56.643831 31872 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786334794676589-31872-0/minicluster-data/ts-0/wal/instance
I20260810 04:06:56.644059 31872 external_mini_cluster.cc:1428] Setting key 32c5cdd8986b8a95416a1400aaa58958
Aug 10 04:06:56 dist-test-slave-8w9v krb5kdc[3820](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334816, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.1@KRBTEST.COM for kudu/127.31.32.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.31.32.2@KRBTEST.COM; defaulting to no policy
I20260810 04:06:56.653738 3932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:56.643785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.1:57279 (local address 127.31.32.62:43109)
0810 04:06:56.643879 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0810 04:06:56.643881 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:06:56.644196 (+ 315us) server_negotiation.cc:407] Connection header received
0810 04:06:56.644868 (+ 672us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:06:56.644868 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:06:56.644881 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:06:56.644922 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:06:56.645905 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:56.646691 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:56.648262 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:56.648457 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:56.648510 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:06:56.650922 (+ 2412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:06:56.650924 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:06:56.650924 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:06:56.650932 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:06:56.652364 (+ 1432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:06:56.652766 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:06:56.652771 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:06:56.652771 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:06:56.652823 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:06:56.653081 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:06:56.653082 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:06:56.653083 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:06:56.653192 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:06:56.653266 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0810 04:06:56.653566 (+ 300us) server_negotiation.cc:299] Negotiation successful
0810 04:06:56.653645 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 04:06:56.653932 4076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:56.643762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:43109 (local address 127.31.32.1:57279)
0810 04:06:56.643984 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0810 04:06:56.644052 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0810 04:06:56.644756 (+ 704us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:06:56.645012 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:06:56.645016 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:06:56.645298 (+ 282us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:06:56.645794 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:06:56.645799 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:06:56.646797 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:06:56.646799 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:06:56.648182 (+ 1383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:06:56.648185 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:06:56.648540 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:06:56.648541 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:06:56.648688 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:06:56.649189 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:06:56.649199 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:06:56.650821 (+ 1622us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:06:56.652445 (+ 1624us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:06:56.652452 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:06:56.652454 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:06:56.652676 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:06:56.652891 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:06:56.652892 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:06:56.652892 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:06:56.653004 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:06:56.653257 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:06:56.653265 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:06:56.653430 (+ 165us) client_negotiation.cc:769] Sending connection context
0810 04:06:56.653544 (+ 114us) client_negotiation.cc:240] Negotiation successful
0810 04:06:56.653700 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":114,"threads_started":1}
I20260810 04:06:56.654511 4074 heartbeater.cc:347] Connected to a master server at 127.31.32.62:43109
I20260810 04:06:56.654624 4074 heartbeater.cc:464] Registering TS with master...
I20260810 04:06:56.654768 4074 heartbeater.cc:511] Master 127.31.32.62:43109 requested a full tablet report, sending...
I20260810 04:06:56.655301 3871 ts_manager.cc:194] Registered new tserver with Master: 5156b5685cc744378665b53c64889c0d (127.31.32.1:41111)
I20260810 04:06:56.656003 3871 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.32.1@KRBTEST.COM'} at 127.31.32.1:57279
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.32.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.32.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.2.keytab.
Entry for principal kudu/127.31.32.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.2.keytab.
I20260810 04:06:56.671649 31872 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskws89Io/build/release/bin/kudu
/tmp/dist-test-taskws89Io/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786334794676589-31872-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786334794676589-31872-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786334794676589-31872-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786334794676589-31872-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.32.2:0
--local_ip_for_outbound_sockets=127.31.32.2
--webserver_interface=127.31.32.2
--webserver_port=0
--tserver_master_addrs=127.31.32.62:43109
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.2.keytab
--principal=kudu/127.31.32.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.32.20:42433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786334794676589-31872-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786334794676589-31872-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786334794676589-31872-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:06:56.751066 4080 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:06:56.751298 4080 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:06:56.751333 4080 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:06:56.752445 4080 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:06:56.752508 4080 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:06:56.752553 4080 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.32.2
I20260810 04:06:56.753978 4080 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.32.20:42433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786334794676589-31872-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786334794676589-31872-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.2.keytab
--principal=kudu/127.31.32.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.32.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786334794676589-31872-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.32.2
--webserver_port=0
--tserver_master_addrs=127.31.32.62:43109
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4080
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.32.2
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786334794676589-31872-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:06:56.754215 4080 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:06:56.754419 4080 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:06:56.759163 4086 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:06:56.759164 4085 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:06:56.759198 4080 server_base.cc:1034] running on GCE node
W20260810 04:06:56.759164 4088 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:06:56.759619 4080 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:06:56.759888 4080 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:06:56.761052 4080 hybrid_clock.cc:648] HybridClock initialized: now 1786334816761040 us; error 29 us; skew 500 ppm
Aug 10 04:06:56 dist-test-slave-8w9v krb5kdc[3820](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334816, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:06:56.763583 4080 init.cc:377] Logged in from keytab as kudu/127.31.32.2@KRBTEST.COM (short username kudu)
I20260810 04:06:56.763968 4080 fs_manager.cc:359] Metadata directory not provided
I20260810 04:06:56.764039 4080 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:06:56.764201 4080 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:06:56.765268 4080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786334794676589-31872-0/minicluster-data/ts-1/data/instance:
uuid: "95387908ea6e44f7af66433f46189ce6"
format_stamp: "Formatted at 2026-08-10 04:06:56 on dist-test-slave-8w9v"
server_key: "2138b662a145e75d5cabfa12d13beb2c"
server_key_iv: "ba23b0706f71ae2e15791aa9655de3e2"
server_key_version: "encryptionkey@0"
I20260810 04:06:56.765661 4080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786334794676589-31872-0/minicluster-data/ts-1/wal/instance:
uuid: "95387908ea6e44f7af66433f46189ce6"
format_stamp: "Formatted at 2026-08-10 04:06:56 on dist-test-slave-8w9v"
server_key: "2138b662a145e75d5cabfa12d13beb2c"
server_key_iv: "ba23b0706f71ae2e15791aa9655de3e2"
server_key_version: "encryptionkey@0"
I20260810 04:06:56.766942 4080 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 04:06:56.767767 4095 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:06:56.767957 4080 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 04:06:56.768062 4080 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786334794676589-31872-0/minicluster-data/ts-1/data,/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786334794676589-31872-0/minicluster-data/ts-1/wal
uuid: "95387908ea6e44f7af66433f46189ce6"
format_stamp: "Formatted at 2026-08-10 04:06:56 on dist-test-slave-8w9v"
server_key: "2138b662a145e75d5cabfa12d13beb2c"
server_key_iv: "ba23b0706f71ae2e15791aa9655de3e2"
server_key_version: "encryptionkey@0"
I20260810 04:06:56.768172 4080 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786334794676589-31872-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786334794676589-31872-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786334794676589-31872-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:06:56.781692 4080 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:06:56.783960 4080 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:06:56.784165 4080 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:06:56.784416 4080 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:06:56.784757 4080 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:06:56.784811 4080 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:06:56.784843 4080 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:06:56.784874 4080 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:06:56.790488 4080 rpc_server.cc:307] RPC server started. Bound to: 127.31.32.2:38783
I20260810 04:06:56.790539 4217 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.32.2:38783 every 8 connection(s)
I20260810 04:06:56.790787 4080 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786334794676589-31872-0/minicluster-data/ts-1/data/info.pb
I20260810 04:06:56.796515 31872 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskws89Io/build/release/bin/kudu as pid 4080
I20260810 04:06:56.796622 31872 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786334794676589-31872-0/minicluster-data/ts-1/wal/instance
Aug 10 04:06:56 dist-test-slave-8w9v krb5kdc[3820](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334816, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.2@KRBTEST.COM for kudu/127.31.32.62@KRBTEST.COM
I20260810 04:06:56.796844 31872 external_mini_cluster.cc:1428] Setting key 0b129c488b6fcd777681d038fb11c106
I20260810 04:06:56.800341 3932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:56.791599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.2:56731 (local address 127.31.32.62:43109)
0810 04:06:56.791654 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0810 04:06:56.791656 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:06:56.791727 (+ 71us) server_negotiation.cc:407] Connection header received
0810 04:06:56.792234 (+ 507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:06:56.792235 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:06:56.792250 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:06:56.792323 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:06:56.793195 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:56.793677 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:56.794598 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:56.794785 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:56.794849 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:06:56.797208 (+ 2359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:06:56.797210 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:06:56.797211 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:06:56.797218 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:06:56.798933 (+ 1715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:06:56.799295 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:06:56.799296 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:06:56.799297 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:06:56.799358 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:06:56.799673 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:06:56.799674 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:06:56.799674 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:06:56.799776 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:06:56.799829 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0810 04:06:56.800181 (+ 352us) server_negotiation.cc:299] Negotiation successful
0810 04:06:56.800259 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260810 04:06:56.800561 4220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:56.791443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:43109 (local address 127.31.32.2:56731)
0810 04:06:56.791602 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0810 04:06:56.791643 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 04:06:56.792089 (+ 446us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:06:56.792383 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:06:56.792386 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:06:56.792623 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:06:56.793082 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:06:56.793087 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:06:56.793761 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:06:56.793761 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:06:56.794511 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:06:56.794513 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:06:56.794867 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:06:56.794868 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:06:56.794959 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:06:56.795446 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:06:56.795454 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:06:56.797118 (+ 1664us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:06:56.799013 (+ 1895us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:06:56.799019 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:06:56.799021 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:06:56.799218 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:06:56.799447 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:06:56.799448 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:06:56.799448 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:06:56.799595 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:06:56.799841 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:06:56.799847 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:06:56.800003 (+ 156us) client_negotiation.cc:769] Sending connection context
0810 04:06:56.800116 (+ 113us) client_negotiation.cc:240] Negotiation successful
0810 04:06:56.800304 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":90,"threads_started":1}
I20260810 04:06:56.801038 4218 heartbeater.cc:347] Connected to a master server at 127.31.32.62:43109
I20260810 04:06:56.801138 4218 heartbeater.cc:464] Registering TS with master...
I20260810 04:06:56.801304 4218 heartbeater.cc:511] Master 127.31.32.62:43109 requested a full tablet report, sending...
I20260810 04:06:56.801663 3871 ts_manager.cc:194] Registered new tserver with Master: 95387908ea6e44f7af66433f46189ce6 (127.31.32.2:38783)
I20260810 04:06:56.802039 3871 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.32.2@KRBTEST.COM'} at 127.31.32.2:56731
WARNING: no policy specified for kudu/127.31.32.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.32.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.32.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.3.keytab.
Entry for principal kudu/127.31.32.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.3.keytab.
I20260810 04:06:56.823153 31872 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskws89Io/build/release/bin/kudu
/tmp/dist-test-taskws89Io/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786334794676589-31872-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786334794676589-31872-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786334794676589-31872-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786334794676589-31872-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.32.3:0
--local_ip_for_outbound_sockets=127.31.32.3
--webserver_interface=127.31.32.3
--webserver_port=0
--tserver_master_addrs=127.31.32.62:43109
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.3.keytab
--principal=kudu/127.31.32.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.32.20:42433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786334794676589-31872-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786334794676589-31872-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786334794676589-31872-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:06:56.901824 4223 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:06:56.902010 4223 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:06:56.902052 4223 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:06:56.903105 4223 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:06:56.903136 4223 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:06:56.903160 4223 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.32.3
I20260810 04:06:56.904521 4223 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.32.20:42433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786334794676589-31872-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786334794676589-31872-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.3.keytab
--principal=kudu/127.31.32.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.32.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786334794676589-31872-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.32.3
--webserver_port=0
--tserver_master_addrs=127.31.32.62:43109
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4223
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.32.3
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786334794676589-31872-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:06:56.904686 4223 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:06:56.904852 4223 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:06:56.909664 4228 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:06:56.909646 4231 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:06:56.909667 4229 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:06:56.909929 4223 server_base.cc:1034] running on GCE node
I20260810 04:06:56.910118 4223 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:06:56.910310 4223 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:06:56.911445 4223 hybrid_clock.cc:648] HybridClock initialized: now 1786334816911436 us; error 25 us; skew 500 ppm
Aug 10 04:06:56 dist-test-slave-8w9v krb5kdc[3820](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334816, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:06:56.913973 4223 init.cc:377] Logged in from keytab as kudu/127.31.32.3@KRBTEST.COM (short username kudu)
I20260810 04:06:56.914543 4223 fs_manager.cc:359] Metadata directory not provided
I20260810 04:06:56.914629 4223 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:06:56.914736 4223 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:06:56.915691 4223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786334794676589-31872-0/minicluster-data/ts-2/data/instance:
uuid: "62b0c41122d541a6a78967010e6652fb"
format_stamp: "Formatted at 2026-08-10 04:06:56 on dist-test-slave-8w9v"
server_key: "8624e1b048114b2940e865c9de0b9bbc"
server_key_iv: "1eef219c9186791ff6b482dbdbf1f995"
server_key_version: "encryptionkey@0"
I20260810 04:06:56.916020 4223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786334794676589-31872-0/minicluster-data/ts-2/wal/instance:
uuid: "62b0c41122d541a6a78967010e6652fb"
format_stamp: "Formatted at 2026-08-10 04:06:56 on dist-test-slave-8w9v"
server_key: "8624e1b048114b2940e865c9de0b9bbc"
server_key_iv: "1eef219c9186791ff6b482dbdbf1f995"
server_key_version: "encryptionkey@0"
I20260810 04:06:56.917392 4223 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260810 04:06:56.918138 4238 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:06:56.918437 4223 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 04:06:56.918520 4223 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786334794676589-31872-0/minicluster-data/ts-2/data,/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786334794676589-31872-0/minicluster-data/ts-2/wal
uuid: "62b0c41122d541a6a78967010e6652fb"
format_stamp: "Formatted at 2026-08-10 04:06:56 on dist-test-slave-8w9v"
server_key: "8624e1b048114b2940e865c9de0b9bbc"
server_key_iv: "1eef219c9186791ff6b482dbdbf1f995"
server_key_version: "encryptionkey@0"
I20260810 04:06:56.918577 4223 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786334794676589-31872-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786334794676589-31872-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786334794676589-31872-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:06:56.927588 4223 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:06:56.929416 4223 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:06:56.929517 4223 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:06:56.929674 4223 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:06:56.929993 4223 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:06:56.930027 4223 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:06:56.930047 4223 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:06:56.930097 4223 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:06:56.935809 4223 rpc_server.cc:307] RPC server started. Bound to: 127.31.32.3:45413
I20260810 04:06:56.935938 4359 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.32.3:45413 every 8 connection(s)
I20260810 04:06:56.936196 4223 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786334794676589-31872-0/minicluster-data/ts-2/data/info.pb
I20260810 04:06:56.937633 31872 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskws89Io/build/release/bin/kudu as pid 4223
I20260810 04:06:56.937741 31872 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786334794676589-31872-0/minicluster-data/ts-2/wal/instance
I20260810 04:06:56.937933 31872 external_mini_cluster.cc:1428] Setting key ac0ecb9a623b61036ac24fe3f421b196
Aug 10 04:06:56 dist-test-slave-8w9v krb5kdc[3820](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334816, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.3@KRBTEST.COM for kudu/127.31.32.62@KRBTEST.COM
I20260810 04:06:56.944983 3932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:56.936921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.3:55413 (local address 127.31.32.62:43109)
0810 04:06:56.936969 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0810 04:06:56.936971 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:06:56.937200 (+ 229us) server_negotiation.cc:407] Connection header received
0810 04:06:56.937646 (+ 446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:06:56.937648 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:06:56.937661 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:06:56.937734 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:06:56.938965 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:56.939487 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:56.940514 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:56.940696 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:56.940764 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:06:56.942764 (+ 2000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:06:56.942765 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:06:56.942765 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:06:56.942772 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:06:56.943948 (+ 1176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:06:56.944323 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:06:56.944323 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:06:56.944324 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:06:56.944370 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:06:56.944555 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:06:56.944555 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:06:56.944555 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:06:56.944642 (+ 87us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:06:56.944685 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0810 04:06:56.944874 (+ 189us) server_negotiation.cc:299] Negotiation successful
0810 04:06:56.944925 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260810 04:06:56.945199 4362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:56.936897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:43109 (local address 127.31.32.3:55413)
0810 04:06:56.937067 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0810 04:06:56.937100 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0810 04:06:56.937552 (+ 452us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:06:56.937861 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:06:56.937865 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:06:56.938214 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:06:56.938891 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:06:56.938897 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:06:56.939567 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:06:56.939568 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:06:56.940442 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:06:56.940444 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:06:56.940781 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:06:56.940782 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:06:56.940874 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:06:56.941369 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:06:56.941376 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:06:56.942668 (+ 1292us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:06:56.944016 (+ 1348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:06:56.944022 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:06:56.944023 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:06:56.944259 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:06:56.944424 (+ 165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:06:56.944424 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:06:56.944424 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:06:56.944511 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:06:56.944688 (+ 177us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:06:56.944691 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:06:56.944786 (+ 95us) client_negotiation.cc:769] Sending connection context
0810 04:06:56.944862 (+ 76us) client_negotiation.cc:240] Negotiation successful
0810 04:06:56.944964 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":94,"threads_started":1}
I20260810 04:06:56.945753 4360 heartbeater.cc:347] Connected to a master server at 127.31.32.62:43109
I20260810 04:06:56.945866 4360 heartbeater.cc:464] Registering TS with master...
I20260810 04:06:56.946022 4360 heartbeater.cc:511] Master 127.31.32.62:43109 requested a full tablet report, sending...
I20260810 04:06:56.946369 3871 ts_manager.cc:194] Registered new tserver with Master: 62b0c41122d541a6a78967010e6652fb (127.31.32.3:45413)
I20260810 04:06:56.946746 3871 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.32.3@KRBTEST.COM'} at 127.31.32.3:55413
I20260810 04:06:56.949641 31872 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 04:06:56.956282 3932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:56.950912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59544 (local address 127.31.32.62:43109)
0810 04:06:56.950965 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0810 04:06:56.950966 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 04:06:56.951046 (+ 80us) server_negotiation.cc:407] Connection header received
0810 04:06:56.951094 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:06:56.951095 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:06:56.951102 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:06:56.951149 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:06:56.951652 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:56.952081 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:56.952828 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:56.953004 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:56.953056 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:06:56.954151 (+ 1095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:06:56.954152 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:06:56.954152 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:06:56.954156 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:06:56.955234 (+ 1078us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:06:56.955525 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:06:56.955526 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:06:56.955526 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:06:56.955573 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:06:56.955754 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:06:56.955755 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:06:56.955755 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:06:56.955916 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:06:56.955990 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0810 04:06:56.956089 (+ 99us) server_negotiation.cc:299] Negotiation successful
0810 04:06:56.956198 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260810 04:06:56.957687 3871 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59544:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 04:06:56.959158 3871 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 04:06:56.965538 4368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:56.961205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.2:38783 (local address 127.0.0.1:47132)
0810 04:06:56.961588 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0810 04:06:56.961595 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0810 04:06:56.961639 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:06:56.961866 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:06:56.961868 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:06:56.961873 (+ 5us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:06:56.961955 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:06:56.961956 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:06:56.963380 (+ 1424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:06:56.963381 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:06:56.964454 (+ 1073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:06:56.964456 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:06:56.965215 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:06:56.965216 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:06:56.965292 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:06:56.965305 (+ 13us) client_negotiation.cc:769] Sending connection context
0810 04:06:56.965411 (+ 106us) client_negotiation.cc:240] Negotiation successful
0810 04:06:56.965468 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":376,"mutex_wait_us":4,"thread_start_us":51,"threads_started":1}
I20260810 04:06:56.965547 4373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:56.961437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47132 (local address 127.31.32.2:38783)
0810 04:06:56.961605 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0810 04:06:56.961609 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 04:06:56.961663 (+ 54us) server_negotiation.cc:407] Connection header received
0810 04:06:56.961710 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:06:56.961712 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:06:56.961729 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:06:56.961819 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:06:56.962199 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:56.963269 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:56.964534 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:56.965094 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:56.965213 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:06:56.965275 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0810 04:06:56.965429 (+ 154us) server_negotiation.cc:299] Negotiation successful
0810 04:06:56.965468 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":74,"threads_started":1}
I20260810 04:06:56.965691 4369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:56.961190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.3:45413 (local address 127.0.0.1:38230)
0810 04:06:56.961373 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0810 04:06:56.961411 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0810 04:06:56.961475 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:06:56.962157 (+ 682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:06:56.962158 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:06:56.962160 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:06:56.962193 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:06:56.962194 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:06:56.963399 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:06:56.963400 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:06:56.964358 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:06:56.964361 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:06:56.965312 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:06:56.965312 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:06:56.965496 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:06:56.965503 (+ 7us) client_negotiation.cc:769] Sending connection context
0810 04:06:56.965558 (+ 55us) client_negotiation.cc:240] Negotiation successful
0810 04:06:56.965616 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"mutex_wait_us":133,"thread_start_us":56,"threads_started":1}
I20260810 04:06:56.965801 4371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:56.961313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42936 (local address 127.31.32.1:41111)
0810 04:06:56.961708 (+ 395us) server_negotiation.cc:206] Beginning negotiation
0810 04:06:56.961711 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:06:56.961720 (+ 9us) server_negotiation.cc:407] Connection header received
0810 04:06:56.961757 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:06:56.961759 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:06:56.961773 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:06:56.961860 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:06:56.962049 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:56.962943 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:56.964432 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:56.965125 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:56.965196 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:06:56.965261 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0810 04:06:56.965662 (+ 401us) server_negotiation.cc:299] Negotiation successful
0810 04:06:56.965705 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":81,"threads_started":1}
I20260810 04:06:56.965888 4370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:56.961191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.1:41111 (local address 127.0.0.1:42936)
0810 04:06:56.961480 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0810 04:06:56.961489 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 04:06:56.961528 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:06:56.961866 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:06:56.961868 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:06:56.961870 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:06:56.961964 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:06:56.961965 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:06:56.963045 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:06:56.963046 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:06:56.964343 (+ 1297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:06:56.964346 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:06:56.965297 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:06:56.965298 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:06:56.965496 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:06:56.965503 (+ 7us) client_negotiation.cc:769] Sending connection context
0810 04:06:56.965558 (+ 55us) client_negotiation.cc:240] Negotiation successful
0810 04:06:56.965616 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"mutex_wait_us":32,"thread_start_us":51,"threads_started":1}
I20260810 04:06:56.966114 4372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:56.961415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38230 (local address 127.31.32.3:45413)
0810 04:06:56.962008 (+ 593us) server_negotiation.cc:206] Beginning negotiation
0810 04:06:56.962012 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 04:06:56.962021 (+ 9us) server_negotiation.cc:407] Connection header received
0810 04:06:56.962055 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:06:56.962057 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:06:56.962071 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:06:56.962147 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:06:56.962252 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:56.963297 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:56.964459 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:56.965188 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:56.965241 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:06:56.965285 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0810 04:06:56.965978 (+ 693us) server_negotiation.cc:299] Negotiation successful
0810 04:06:56.966040 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":548,"thread_start_us":79,"threads_started":1}
I20260810 04:06:56.966698 4286 tablet_service.cc:1467] Processing CreateTablet for tablet 7772bb3973ae4596b292a313778ae62c (DEFAULT_TABLE table=test-table [id=f49ecc18e8f842a29a3e85a6cfde4f2b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:06:56.966698 4005 tablet_service.cc:1467] Processing CreateTablet for tablet 7772bb3973ae4596b292a313778ae62c (DEFAULT_TABLE table=test-table [id=f49ecc18e8f842a29a3e85a6cfde4f2b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:06:56.967046 4005 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7772bb3973ae4596b292a313778ae62c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:06:56.967046 4286 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7772bb3973ae4596b292a313778ae62c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:06:56.968710 4147 tablet_service.cc:1467] Processing CreateTablet for tablet 7772bb3973ae4596b292a313778ae62c (DEFAULT_TABLE table=test-table [id=f49ecc18e8f842a29a3e85a6cfde4f2b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:06:56.968948 4147 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7772bb3973ae4596b292a313778ae62c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:06:56.970381 4375 tablet_bootstrap.cc:492] T 7772bb3973ae4596b292a313778ae62c P 62b0c41122d541a6a78967010e6652fb: Bootstrap starting.
I20260810 04:06:56.971139 4375 tablet_bootstrap.cc:654] T 7772bb3973ae4596b292a313778ae62c P 62b0c41122d541a6a78967010e6652fb: Neither blocks nor log segments found. Creating new log.
I20260810 04:06:56.971870 4377 tablet_bootstrap.cc:492] T 7772bb3973ae4596b292a313778ae62c P 95387908ea6e44f7af66433f46189ce6: Bootstrap starting.
I20260810 04:06:56.971907 4375 log.cc:824] T 7772bb3973ae4596b292a313778ae62c P 62b0c41122d541a6a78967010e6652fb: Log is configured to *not* fsync() on all Append() calls
I20260810 04:06:56.972263 4374 tablet_bootstrap.cc:492] T 7772bb3973ae4596b292a313778ae62c P 5156b5685cc744378665b53c64889c0d: Bootstrap starting.
I20260810 04:06:56.972533 4377 tablet_bootstrap.cc:654] T 7772bb3973ae4596b292a313778ae62c P 95387908ea6e44f7af66433f46189ce6: Neither blocks nor log segments found. Creating new log.
I20260810 04:06:56.972687 4375 tablet_bootstrap.cc:492] T 7772bb3973ae4596b292a313778ae62c P 62b0c41122d541a6a78967010e6652fb: No bootstrap required, opened a new log
I20260810 04:06:56.972752 4375 ts_tablet_manager.cc:1397] T 7772bb3973ae4596b292a313778ae62c P 62b0c41122d541a6a78967010e6652fb: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 04:06:56.972922 4374 tablet_bootstrap.cc:654] T 7772bb3973ae4596b292a313778ae62c P 5156b5685cc744378665b53c64889c0d: Neither blocks nor log segments found. Creating new log.
I20260810 04:06:56.973134 4377 log.cc:824] T 7772bb3973ae4596b292a313778ae62c P 95387908ea6e44f7af66433f46189ce6: Log is configured to *not* fsync() on all Append() calls
I20260810 04:06:56.973517 4374 log.cc:824] T 7772bb3973ae4596b292a313778ae62c P 5156b5685cc744378665b53c64889c0d: Log is configured to *not* fsync() on all Append() calls
I20260810 04:06:56.973908 4377 tablet_bootstrap.cc:492] T 7772bb3973ae4596b292a313778ae62c P 95387908ea6e44f7af66433f46189ce6: No bootstrap required, opened a new log
I20260810 04:06:56.974009 4377 ts_tablet_manager.cc:1397] T 7772bb3973ae4596b292a313778ae62c P 95387908ea6e44f7af66433f46189ce6: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 04:06:56.974393 4374 tablet_bootstrap.cc:492] T 7772bb3973ae4596b292a313778ae62c P 5156b5685cc744378665b53c64889c0d: No bootstrap required, opened a new log
I20260810 04:06:56.974464 4374 ts_tablet_manager.cc:1397] T 7772bb3973ae4596b292a313778ae62c P 5156b5685cc744378665b53c64889c0d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 04:06:56.975016 4375 raft_consensus.cc:348] T 7772bb3973ae4596b292a313778ae62c P 62b0c41122d541a6a78967010e6652fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62b0c41122d541a6a78967010e6652fb" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 45413 } } peers { permanent_uuid: "5156b5685cc744378665b53c64889c0d" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 41111 } } peers { permanent_uuid: "95387908ea6e44f7af66433f46189ce6" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 38783 } }
I20260810 04:06:56.975184 4375 raft_consensus.cc:374] T 7772bb3973ae4596b292a313778ae62c P 62b0c41122d541a6a78967010e6652fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:06:56.975220 4375 raft_consensus.cc:729] T 7772bb3973ae4596b292a313778ae62c P 62b0c41122d541a6a78967010e6652fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62b0c41122d541a6a78967010e6652fb, State: Initialized, Role: FOLLOWER
I20260810 04:06:56.975346 4375 consensus_queue.cc:260] T 7772bb3973ae4596b292a313778ae62c P 62b0c41122d541a6a78967010e6652fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62b0c41122d541a6a78967010e6652fb" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 45413 } } peers { permanent_uuid: "5156b5685cc744378665b53c64889c0d" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 41111 } } peers { permanent_uuid: "95387908ea6e44f7af66433f46189ce6" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 38783 } }
I20260810 04:06:56.975610 4375 ts_tablet_manager.cc:1428] T 7772bb3973ae4596b292a313778ae62c P 62b0c41122d541a6a78967010e6652fb: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 04:06:56.975729 4360 heartbeater.cc:503] Master 127.31.32.62:43109 was elected leader, sending a full tablet report...
I20260810 04:06:56.975611 4377 raft_consensus.cc:348] T 7772bb3973ae4596b292a313778ae62c P 95387908ea6e44f7af66433f46189ce6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62b0c41122d541a6a78967010e6652fb" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 45413 } } peers { permanent_uuid: "5156b5685cc744378665b53c64889c0d" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 41111 } } peers { permanent_uuid: "95387908ea6e44f7af66433f46189ce6" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 38783 } }
I20260810 04:06:56.975776 4377 raft_consensus.cc:374] T 7772bb3973ae4596b292a313778ae62c P 95387908ea6e44f7af66433f46189ce6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:06:56.975821 4377 raft_consensus.cc:729] T 7772bb3973ae4596b292a313778ae62c P 95387908ea6e44f7af66433f46189ce6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95387908ea6e44f7af66433f46189ce6, State: Initialized, Role: FOLLOWER
I20260810 04:06:56.975950 4377 consensus_queue.cc:260] T 7772bb3973ae4596b292a313778ae62c P 95387908ea6e44f7af66433f46189ce6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62b0c41122d541a6a78967010e6652fb" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 45413 } } peers { permanent_uuid: "5156b5685cc744378665b53c64889c0d" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 41111 } } peers { permanent_uuid: "95387908ea6e44f7af66433f46189ce6" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 38783 } }
I20260810 04:06:56.976270 4218 heartbeater.cc:503] Master 127.31.32.62:43109 was elected leader, sending a full tablet report...
I20260810 04:06:56.976251 4377 ts_tablet_manager.cc:1428] T 7772bb3973ae4596b292a313778ae62c P 95387908ea6e44f7af66433f46189ce6: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260810 04:06:56.977028 4374 raft_consensus.cc:348] T 7772bb3973ae4596b292a313778ae62c P 5156b5685cc744378665b53c64889c0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62b0c41122d541a6a78967010e6652fb" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 45413 } } peers { permanent_uuid: "5156b5685cc744378665b53c64889c0d" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 41111 } } peers { permanent_uuid: "95387908ea6e44f7af66433f46189ce6" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 38783 } }
I20260810 04:06:56.977157 4374 raft_consensus.cc:374] T 7772bb3973ae4596b292a313778ae62c P 5156b5685cc744378665b53c64889c0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:06:56.977177 4374 raft_consensus.cc:729] T 7772bb3973ae4596b292a313778ae62c P 5156b5685cc744378665b53c64889c0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5156b5685cc744378665b53c64889c0d, State: Initialized, Role: FOLLOWER
I20260810 04:06:56.977277 4374 consensus_queue.cc:260] T 7772bb3973ae4596b292a313778ae62c P 5156b5685cc744378665b53c64889c0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62b0c41122d541a6a78967010e6652fb" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 45413 } } peers { permanent_uuid: "5156b5685cc744378665b53c64889c0d" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 41111 } } peers { permanent_uuid: "95387908ea6e44f7af66433f46189ce6" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 38783 } }
I20260810 04:06:56.977571 4374 ts_tablet_manager.cc:1428] T 7772bb3973ae4596b292a313778ae62c P 5156b5685cc744378665b53c64889c0d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 04:06:56.977583 4074 heartbeater.cc:503] Master 127.31.32.62:43109 was elected leader, sending a full tablet report...
W20260810 04:06:57.041275 4219 tablet.cc:2369] T 7772bb3973ae4596b292a313778ae62c P 95387908ea6e44f7af66433f46189ce6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 04:06:57.102708 4382 raft_consensus.cc:482] T 7772bb3973ae4596b292a313778ae62c P 95387908ea6e44f7af66433f46189ce6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 04:06:57.102847 4382 raft_consensus.cc:504] T 7772bb3973ae4596b292a313778ae62c P 95387908ea6e44f7af66433f46189ce6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62b0c41122d541a6a78967010e6652fb" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 45413 } } peers { permanent_uuid: "5156b5685cc744378665b53c64889c0d" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 41111 } } peers { permanent_uuid: "95387908ea6e44f7af66433f46189ce6" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 38783 } }
I20260810 04:06:57.103335 4382 leader_election.cc:288] T 7772bb3973ae4596b292a313778ae62c P 95387908ea6e44f7af66433f46189ce6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62b0c41122d541a6a78967010e6652fb (127.31.32.3:45413), 5156b5685cc744378665b53c64889c0d (127.31.32.1:41111)
I20260810 04:06:57.106487 4372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:57.103657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.2:47693 (local address 127.31.32.3:45413)
0810 04:06:57.103708 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0810 04:06:57.103710 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:06:57.103722 (+ 12us) server_negotiation.cc:407] Connection header received
0810 04:06:57.103757 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:06:57.103758 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:06:57.103771 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:06:57.103855 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:06:57.104113 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:57.104703 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:57.105682 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:57.106082 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:57.106169 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:06:57.106180 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0810 04:06:57.106362 (+ 182us) server_negotiation.cc:299] Negotiation successful
0810 04:06:57.106417 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260810 04:06:57.106465 4220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:57.103551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.3:45413 (local address 127.31.32.2:47693)
0810 04:06:57.103620 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0810 04:06:57.103633 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 04:06:57.103683 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:06:57.103914 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:06:57.103915 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:06:57.103917 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:06:57.103999 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:06:57.104002 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:06:57.104784 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:06:57.104785 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:06:57.105586 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:06:57.105588 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:06:57.106169 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:06:57.106170 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:06:57.106275 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:06:57.106279 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 04:06:57.106337 (+ 58us) client_negotiation.cc:240] Negotiation successful
0810 04:06:57.106379 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260810 04:06:57.106557 4371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:57.103637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.2:44491 (local address 127.31.32.1:41111)
0810 04:06:57.103685 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0810 04:06:57.103688 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:06:57.103900 (+ 212us) server_negotiation.cc:407] Connection header received
0810 04:06:57.103947 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:06:57.103948 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:06:57.103961 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:06:57.104019 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:06:57.104292 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:57.104783 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:57.105761 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:57.106199 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:57.106274 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:06:57.106285 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0810 04:06:57.106443 (+ 158us) server_negotiation.cc:299] Negotiation successful
0810 04:06:57.106484 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260810 04:06:57.106571 4384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:57.103551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.1:41111 (local address 127.31.32.2:44491)
0810 04:06:57.103771 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0810 04:06:57.103802 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0810 04:06:57.103864 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:06:57.104107 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:06:57.104108 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:06:57.104110 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:06:57.104207 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:06:57.104209 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:06:57.104886 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:06:57.104886 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:06:57.105664 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:06:57.105666 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:06:57.106284 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:06:57.106284 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:06:57.106366 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:06:57.106369 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 04:06:57.106419 (+ 50us) client_negotiation.cc:240] Negotiation successful
0810 04:06:57.106448 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"mutex_wait_us":26,"thread_start_us":124,"threads_started":1}
I20260810 04:06:57.106923 4309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7772bb3973ae4596b292a313778ae62c" candidate_uuid: "95387908ea6e44f7af66433f46189ce6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62b0c41122d541a6a78967010e6652fb" is_pre_election: true
I20260810 04:06:57.106969 4026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7772bb3973ae4596b292a313778ae62c" candidate_uuid: "95387908ea6e44f7af66433f46189ce6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5156b5685cc744378665b53c64889c0d" is_pre_election: true
I20260810 04:06:57.107113 4309 raft_consensus.cc:2454] T 7772bb3973ae4596b292a313778ae62c P 62b0c41122d541a6a78967010e6652fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95387908ea6e44f7af66433f46189ce6 in term 0.
I20260810 04:06:57.107136 4026 raft_consensus.cc:2454] T 7772bb3973ae4596b292a313778ae62c P 5156b5685cc744378665b53c64889c0d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95387908ea6e44f7af66433f46189ce6 in term 0.
I20260810 04:06:57.107389 4099 leader_election.cc:302] T 7772bb3973ae4596b292a313778ae62c P 95387908ea6e44f7af66433f46189ce6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5156b5685cc744378665b53c64889c0d, 95387908ea6e44f7af66433f46189ce6; no voters:
I20260810 04:06:57.107559 4382 raft_consensus.cc:2781] T 7772bb3973ae4596b292a313778ae62c P 95387908ea6e44f7af66433f46189ce6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 04:06:57.107620 4382 raft_consensus.cc:482] T 7772bb3973ae4596b292a313778ae62c P 95387908ea6e44f7af66433f46189ce6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 04:06:57.107645 4382 raft_consensus.cc:3037] T 7772bb3973ae4596b292a313778ae62c P 95387908ea6e44f7af66433f46189ce6 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:06:57.108309 4382 raft_consensus.cc:504] T 7772bb3973ae4596b292a313778ae62c P 95387908ea6e44f7af66433f46189ce6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62b0c41122d541a6a78967010e6652fb" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 45413 } } peers { permanent_uuid: "5156b5685cc744378665b53c64889c0d" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 41111 } } peers { permanent_uuid: "95387908ea6e44f7af66433f46189ce6" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 38783 } }
I20260810 04:06:57.108518 4382 leader_election.cc:288] T 7772bb3973ae4596b292a313778ae62c P 95387908ea6e44f7af66433f46189ce6 [CANDIDATE]: Term 1 election: Requested vote from peers 62b0c41122d541a6a78967010e6652fb (127.31.32.3:45413), 5156b5685cc744378665b53c64889c0d (127.31.32.1:41111)
I20260810 04:06:57.108574 4309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7772bb3973ae4596b292a313778ae62c" candidate_uuid: "95387908ea6e44f7af66433f46189ce6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62b0c41122d541a6a78967010e6652fb"
I20260810 04:06:57.108604 4026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7772bb3973ae4596b292a313778ae62c" candidate_uuid: "95387908ea6e44f7af66433f46189ce6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5156b5685cc744378665b53c64889c0d"
I20260810 04:06:57.108644 4309 raft_consensus.cc:3037] T 7772bb3973ae4596b292a313778ae62c P 62b0c41122d541a6a78967010e6652fb [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:06:57.108678 4026 raft_consensus.cc:3037] T 7772bb3973ae4596b292a313778ae62c P 5156b5685cc744378665b53c64889c0d [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:06:57.109427 4026 raft_consensus.cc:2454] T 7772bb3973ae4596b292a313778ae62c P 5156b5685cc744378665b53c64889c0d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95387908ea6e44f7af66433f46189ce6 in term 1.
I20260810 04:06:57.109472 4309 raft_consensus.cc:2454] T 7772bb3973ae4596b292a313778ae62c P 62b0c41122d541a6a78967010e6652fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95387908ea6e44f7af66433f46189ce6 in term 1.
I20260810 04:06:57.109624 4099 leader_election.cc:302] T 7772bb3973ae4596b292a313778ae62c P 95387908ea6e44f7af66433f46189ce6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5156b5685cc744378665b53c64889c0d, 95387908ea6e44f7af66433f46189ce6; no voters:
I20260810 04:06:57.109730 4382 raft_consensus.cc:2781] T 7772bb3973ae4596b292a313778ae62c P 95387908ea6e44f7af66433f46189ce6 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:06:57.109910 4382 raft_consensus.cc:686] T 7772bb3973ae4596b292a313778ae62c P 95387908ea6e44f7af66433f46189ce6 [term 1 LEADER]: Becoming Leader. State: Replica: 95387908ea6e44f7af66433f46189ce6, State: Running, Role: LEADER
I20260810 04:06:57.110028 4382 consensus_queue.cc:237] T 7772bb3973ae4596b292a313778ae62c P 95387908ea6e44f7af66433f46189ce6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62b0c41122d541a6a78967010e6652fb" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 45413 } } peers { permanent_uuid: "5156b5685cc744378665b53c64889c0d" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 41111 } } peers { permanent_uuid: "95387908ea6e44f7af66433f46189ce6" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 38783 } }
I20260810 04:06:57.110865 3870 catalog_manager.cc:5495] T 7772bb3973ae4596b292a313778ae62c P 95387908ea6e44f7af66433f46189ce6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 95387908ea6e44f7af66433f46189ce6 (127.31.32.2). New cstate: current_term: 1 leader_uuid: "95387908ea6e44f7af66433f46189ce6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62b0c41122d541a6a78967010e6652fb" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 45413 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5156b5685cc744378665b53c64889c0d" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 41111 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "95387908ea6e44f7af66433f46189ce6" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 38783 } health_report { overall_health: HEALTHY } } }
W20260810 04:06:57.143752 4075 tablet.cc:2369] T 7772bb3973ae4596b292a313778ae62c P 5156b5685cc744378665b53c64889c0d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 04:06:57.149049 4373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:57.146661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47148 (local address 127.31.32.2:38783)
0810 04:06:57.146724 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0810 04:06:57.146727 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:06:57.146736 (+ 9us) server_negotiation.cc:407] Connection header received
0810 04:06:57.146810 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:06:57.146811 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:06:57.146834 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:06:57.146901 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 04:06:57.147047 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:57.147473 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:57.148267 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:57.148449 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:57.148532 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:06:57.148707 (+ 175us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 04:06:57.148803 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 04:06:57.148899 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0810 04:06:57.148921 (+ 22us) server_negotiation.cc:299] Negotiation successful
0810 04:06:57.148942 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260810 04:06:57.150808 4026 raft_consensus.cc:1261] T 7772bb3973ae4596b292a313778ae62c P 5156b5685cc744378665b53c64889c0d [term 1 FOLLOWER]: Refusing update from remote peer 95387908ea6e44f7af66433f46189ce6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:06:57.150810 4309 raft_consensus.cc:1261] T 7772bb3973ae4596b292a313778ae62c P 62b0c41122d541a6a78967010e6652fb [term 1 FOLLOWER]: Refusing update from remote peer 95387908ea6e44f7af66433f46189ce6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:06:57.151168 4382 consensus_queue.cc:1035] T 7772bb3973ae4596b292a313778ae62c P 95387908ea6e44f7af66433f46189ce6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62b0c41122d541a6a78967010e6652fb" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 45413 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:06:57.151327 4385 consensus_queue.cc:1035] T 7772bb3973ae4596b292a313778ae62c P 95387908ea6e44f7af66433f46189ce6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5156b5685cc744378665b53c64889c0d" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 41111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:06:57.154490 4389 mvcc.cc:205] Tried to move back new op lower bound from 7316827411047133184 to 7316827410883260416. Current Snapshot: MvccSnapshot[applied={T|T < 7316827411047133184 or (T in {7316827411047133184})}]
I20260810 04:06:57.154626 4390 mvcc.cc:205] Tried to move back new op lower bound from 7316827411047133184 to 7316827410883260416. Current Snapshot: MvccSnapshot[applied={T|T < 7316827411047133184 or (T in {7316827411047133184})}]
I20260810 04:06:57.156307 3870 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59544:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 04:06:57.156409 3870 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59544:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 04:06:57.157197 3870 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6f4ed470c94343d29ce287ac5e7a6108: Sending DeleteTablet for 3 replicas of tablet 7772bb3973ae4596b292a313778ae62c
I20260810 04:06:57.157507 4286 tablet_service.cc:1514] Processing DeleteTablet for tablet 7772bb3973ae4596b292a313778ae62c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:06:57 UTC) from {username='kudu', principal='kudu/127.31.32.62@KRBTEST.COM'} at 127.0.0.1:38230
I20260810 04:06:57.157509 4147 tablet_service.cc:1514] Processing DeleteTablet for tablet 7772bb3973ae4596b292a313778ae62c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:06:57 UTC) from {username='kudu', principal='kudu/127.31.32.62@KRBTEST.COM'} at 127.0.0.1:47132
I20260810 04:06:57.157505 4005 tablet_service.cc:1514] Processing DeleteTablet for tablet 7772bb3973ae4596b292a313778ae62c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:06:57 UTC) from {username='kudu', principal='kudu/127.31.32.62@KRBTEST.COM'} at 127.0.0.1:42936
I20260810 04:06:57.157778 4402 tablet_replica.cc:330] T 7772bb3973ae4596b292a313778ae62c P 95387908ea6e44f7af66433f46189ce6: stopping tablet replica
I20260810 04:06:57.157768 4400 tablet_replica.cc:330] T 7772bb3973ae4596b292a313778ae62c P 5156b5685cc744378665b53c64889c0d: stopping tablet replica
I20260810 04:06:57.157861 4402 raft_consensus.cc:2229] T 7772bb3973ae4596b292a313778ae62c P 95387908ea6e44f7af66433f46189ce6 [term 1 LEADER]: Raft consensus shutting down.
I20260810 04:06:57.157861 4400 raft_consensus.cc:2229] T 7772bb3973ae4596b292a313778ae62c P 5156b5685cc744378665b53c64889c0d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 04:06:57.157954 4400 raft_consensus.cc:2258] T 7772bb3973ae4596b292a313778ae62c P 5156b5685cc744378665b53c64889c0d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:06:57.157963 4402 raft_consensus.cc:2258] T 7772bb3973ae4596b292a313778ae62c P 95387908ea6e44f7af66433f46189ce6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:06:57.158094 4401 tablet_replica.cc:330] T 7772bb3973ae4596b292a313778ae62c P 62b0c41122d541a6a78967010e6652fb: stopping tablet replica
I20260810 04:06:57.158195 4401 raft_consensus.cc:2229] T 7772bb3973ae4596b292a313778ae62c P 62b0c41122d541a6a78967010e6652fb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 04:06:57.158262 31872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskws89Io/build/release/bin/kudu with pid 3938
I20260810 04:06:57.158296 4401 raft_consensus.cc:2258] T 7772bb3973ae4596b292a313778ae62c P 62b0c41122d541a6a78967010e6652fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:06:57.158442 4400 ts_tablet_manager.cc:1905] T 7772bb3973ae4596b292a313778ae62c P 5156b5685cc744378665b53c64889c0d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:06:57.160269 4401 ts_tablet_manager.cc:1905] T 7772bb3973ae4596b292a313778ae62c P 62b0c41122d541a6a78967010e6652fb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:06:57.161661 4401 ts_tablet_manager.cc:1918] T 7772bb3973ae4596b292a313778ae62c P 62b0c41122d541a6a78967010e6652fb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 04:06:57.161713 4401 log.cc:1196] T 7772bb3973ae4596b292a313778ae62c P 62b0c41122d541a6a78967010e6652fb: Deleting WAL directory at /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786334794676589-31872-0/minicluster-data/ts-2/wal/wals/7772bb3973ae4596b292a313778ae62c
I20260810 04:06:57.161966 4401 ts_tablet_manager.cc:1939] T 7772bb3973ae4596b292a313778ae62c P 62b0c41122d541a6a78967010e6652fb: Deleting consensus metadata
I20260810 04:06:57.162290 3853 catalog_manager.cc:4841] TS 62b0c41122d541a6a78967010e6652fb (127.31.32.3:45413): tablet 7772bb3973ae4596b292a313778ae62c (table test-table [id=f49ecc18e8f842a29a3e85a6cfde4f2b]) successfully deleted
I20260810 04:06:57.164733 4402 ts_tablet_manager.cc:1905] T 7772bb3973ae4596b292a313778ae62c P 95387908ea6e44f7af66433f46189ce6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:06:57.166409 4402 ts_tablet_manager.cc:1918] T 7772bb3973ae4596b292a313778ae62c P 95387908ea6e44f7af66433f46189ce6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 04:06:57.166481 4402 log.cc:1196] T 7772bb3973ae4596b292a313778ae62c P 95387908ea6e44f7af66433f46189ce6: Deleting WAL directory at /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786334794676589-31872-0/minicluster-data/ts-1/wal/wals/7772bb3973ae4596b292a313778ae62c
W20260810 04:06:57.166615 3855 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.32.1:41111 (error 108)
I20260810 04:06:57.166756 4402 ts_tablet_manager.cc:1939] T 7772bb3973ae4596b292a313778ae62c P 95387908ea6e44f7af66433f46189ce6: Deleting consensus metadata
I20260810 04:06:57.167150 4369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:57.167001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.1:41111 (local address 127.0.0.1:42940)
0810 04:06:57.167057 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0810 04:06:57.167096 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.1:41111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260810 04:06:57.167165 3854 catalog_manager.cc:4841] TS 95387908ea6e44f7af66433f46189ce6 (127.31.32.2:38783): tablet 7772bb3973ae4596b292a313778ae62c (table test-table [id=f49ecc18e8f842a29a3e85a6cfde4f2b]) successfully deleted
W20260810 04:06:57.167276 3855 catalog_manager.cc:4584] TS 5156b5685cc744378665b53c64889c0d (127.31.32.1:41111): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7772bb3973ae4596b292a313778ae62c: Network error: Client connection negotiation failed: client connection to 127.31.32.1:41111: connect: Connection refused (error 111)
I20260810 04:06:57.167588 31872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskws89Io/build/release/bin/kudu with pid 4080
I20260810 04:06:57.173249 31872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskws89Io/build/release/bin/kudu with pid 4223
I20260810 04:06:57.179142 31872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskws89Io/build/release/bin/kudu with pid 3835
2026-08-10T04:06:57Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2025 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-08-10T04:06:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T04:06:57Z Disabled control of system clock
I20260810 04:06:57.209182 31872 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskws89Io/build/release/bin/kudu
/tmp/dist-test-taskws89Io/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786334794676589-31872-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786334794676589-31872-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786334794676589-31872-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786334794676589-31872-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.32.62:39167
--webserver_interface=127.31.32.62
--webserver_port=0
--builtin_ntp_servers=127.31.32.20:43129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.32.62:39167
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260810 04:06:57.291031 4407 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:06:57.291220 4407 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:06:57.291240 4407 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:06:57.292423 4407 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:06:57.292461 4407 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:06:57.292474 4407 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:06:57.292486 4407 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:06:57.292497 4407 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:06:57.293838 4407 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.32.20:43129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786334794676589-31872-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786334794676589-31872-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.32.62:39167
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.32.62:39167
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786334794676589-31872-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.32.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4407
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786334794676589-31872-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:06:57.294075 4407 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:06:57.294395 4407 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:06:57.299217 4415 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:06:57.299268 4413 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:06:57.299242 4412 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:06:57.299715 4407 server_base.cc:1034] running on GCE node
I20260810 04:06:57.299955 4407 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:06:57.300292 4407 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:06:57.301462 4407 hybrid_clock.cc:648] HybridClock initialized: now 1786334817301441 us; error 41 us; skew 500 ppm
I20260810 04:06:57.302798 4407 webserver.cc:460] Webserver started at http://127.31.32.62:44225/ using document root <none> and password file <none>
I20260810 04:06:57.303079 4407 fs_manager.cc:359] Metadata directory not provided
I20260810 04:06:57.303148 4407 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:06:57.303280 4407 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:06:57.304347 4407 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786334794676589-31872-0/minicluster-data/master-0/data/instance:
uuid: "255a721df0514556b482787491b7f228"
format_stamp: "Formatted at 2026-08-10 04:06:57 on dist-test-slave-8w9v"
server_key: "000d9a0fcd80e20333e32e0fb2064dbd"
server_key_iv: "cd3d57483b174452e69ed475062f55b0"
server_key_version: "encryptionkey@0"
I20260810 04:06:57.304700 4407 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786334794676589-31872-0/minicluster-data/master-0/wal/instance:
uuid: "255a721df0514556b482787491b7f228"
format_stamp: "Formatted at 2026-08-10 04:06:57 on dist-test-slave-8w9v"
server_key: "000d9a0fcd80e20333e32e0fb2064dbd"
server_key_iv: "cd3d57483b174452e69ed475062f55b0"
server_key_version: "encryptionkey@0"
I20260810 04:06:57.306016 4407 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 04:06:57.306802 4421 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:06:57.306975 4407 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:06:57.307066 4407 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786334794676589-31872-0/minicluster-data/master-0/data,/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786334794676589-31872-0/minicluster-data/master-0/wal
uuid: "255a721df0514556b482787491b7f228"
format_stamp: "Formatted at 2026-08-10 04:06:57 on dist-test-slave-8w9v"
server_key: "000d9a0fcd80e20333e32e0fb2064dbd"
server_key_iv: "cd3d57483b174452e69ed475062f55b0"
server_key_version: "encryptionkey@0"
I20260810 04:06:57.307190 4407 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786334794676589-31872-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786334794676589-31872-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786334794676589-31872-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:06:57.311511 4407 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:06:57.311797 4407 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:06:57.311909 4407 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:06:57.316107 4407 rpc_server.cc:307] RPC server started. Bound to: 127.31.32.62:39167
I20260810 04:06:57.316521 4407 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786334794676589-31872-0/minicluster-data/master-0/data/info.pb
I20260810 04:06:57.317363 4478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:06:57.317930 4477 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.32.62:39167 every 8 connection(s)
I20260810 04:06:57.320729 4478 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 255a721df0514556b482787491b7f228: Bootstrap starting.
I20260810 04:06:57.321415 4478 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 255a721df0514556b482787491b7f228: Neither blocks nor log segments found. Creating new log.
I20260810 04:06:57.321725 4478 log.cc:824] T 00000000000000000000000000000000 P 255a721df0514556b482787491b7f228: Log is configured to *not* fsync() on all Append() calls
I20260810 04:06:57.322487 4478 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 255a721df0514556b482787491b7f228: No bootstrap required, opened a new log
I20260810 04:06:57.324066 4478 raft_consensus.cc:348] T 00000000000000000000000000000000 P 255a721df0514556b482787491b7f228 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "255a721df0514556b482787491b7f228" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 39167 } }
I20260810 04:06:57.324258 4478 raft_consensus.cc:374] T 00000000000000000000000000000000 P 255a721df0514556b482787491b7f228 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:06:57.324286 4478 raft_consensus.cc:729] T 00000000000000000000000000000000 P 255a721df0514556b482787491b7f228 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 255a721df0514556b482787491b7f228, State: Initialized, Role: FOLLOWER
I20260810 04:06:57.324458 4478 consensus_queue.cc:260] T 00000000000000000000000000000000 P 255a721df0514556b482787491b7f228 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "255a721df0514556b482787491b7f228" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 39167 } }
I20260810 04:06:57.324518 4478 raft_consensus.cc:388] T 00000000000000000000000000000000 P 255a721df0514556b482787491b7f228 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 04:06:57.324572 4478 raft_consensus.cc:482] T 00000000000000000000000000000000 P 255a721df0514556b482787491b7f228 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 04:06:57.324622 4478 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 255a721df0514556b482787491b7f228 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:06:57.325147 4478 raft_consensus.cc:504] T 00000000000000000000000000000000 P 255a721df0514556b482787491b7f228 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "255a721df0514556b482787491b7f228" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 39167 } }
I20260810 04:06:57.325253 4478 leader_election.cc:302] T 00000000000000000000000000000000 P 255a721df0514556b482787491b7f228 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 255a721df0514556b482787491b7f228; no voters:
I20260810 04:06:57.325435 4478 leader_election.cc:288] T 00000000000000000000000000000000 P 255a721df0514556b482787491b7f228 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 04:06:57.325518 4481 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 255a721df0514556b482787491b7f228 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:06:57.325711 4478 sys_catalog.cc:564] T 00000000000000000000000000000000 P 255a721df0514556b482787491b7f228 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:06:57.325740 4481 raft_consensus.cc:686] T 00000000000000000000000000000000 P 255a721df0514556b482787491b7f228 [term 1 LEADER]: Becoming Leader. State: Replica: 255a721df0514556b482787491b7f228, State: Running, Role: LEADER
I20260810 04:06:57.325910 4481 consensus_queue.cc:237] T 00000000000000000000000000000000 P 255a721df0514556b482787491b7f228 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "255a721df0514556b482787491b7f228" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 39167 } }
I20260810 04:06:57.326470 31872 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskws89Io/build/release/bin/kudu as pid 4407
I20260810 04:06:57.326488 4482 sys_catalog.cc:455] T 00000000000000000000000000000000 P 255a721df0514556b482787491b7f228 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "255a721df0514556b482787491b7f228" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "255a721df0514556b482787491b7f228" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 39167 } } }
I20260810 04:06:57.326577 31872 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786334794676589-31872-0/minicluster-data/master-0/wal/instance
I20260810 04:06:57.326567 4482 sys_catalog.cc:458] T 00000000000000000000000000000000 P 255a721df0514556b482787491b7f228 [sys.catalog]: This master's current role is: LEADER
I20260810 04:06:57.326589 4483 sys_catalog.cc:455] T 00000000000000000000000000000000 P 255a721df0514556b482787491b7f228 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 255a721df0514556b482787491b7f228. Latest consensus state: current_term: 1 leader_uuid: "255a721df0514556b482787491b7f228" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "255a721df0514556b482787491b7f228" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 39167 } } }
I20260810 04:06:57.326650 4483 sys_catalog.cc:458] T 00000000000000000000000000000000 P 255a721df0514556b482787491b7f228 [sys.catalog]: This master's current role is: LEADER
I20260810 04:06:57.326783 31872 external_mini_cluster.cc:1428] Setting key 2a27b025e7aac82919c90425982c6797
I20260810 04:06:57.326794 4489 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 04:06:57.327303 4489 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 04:06:57.329161 4489 catalog_manager.cc:1347] Generated new cluster ID: ecd6013087cf4e45b3d45554c54af4cc
I20260810 04:06:57.329222 4489 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 04:06:57.337127 4500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:57.327458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48420 (local address 127.31.32.62:39167)
0810 04:06:57.327925 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0810 04:06:57.327932 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 04:06:57.327957 (+ 25us) server_negotiation.cc:407] Connection header received
0810 04:06:57.328172 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:06:57.328180 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:06:57.328233 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:06:57.328486 (+ 253us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:06:57.336472 (+ 7986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:06:57.336476 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:06:57.336476 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:06:57.336481 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:06:57.336533 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:06:57.336549 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:06:57.336603 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0810 04:06:57.336862 (+ 259us) server_negotiation.cc:299] Negotiation successful
0810 04:06:57.336928 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":90,"threads_started":1}
I20260810 04:06:57.343781 4489 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 04:06:57.343926 4489 catalog_manager.cc:1504] Loading token signing keys...
I20260810 04:06:57.347620 4489 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 255a721df0514556b482787491b7f228: Generated new TSK 0
I20260810 04:06:57.347784 4489 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 04:06:57.349135 31872 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskws89Io/build/release/bin/kudu
/tmp/dist-test-taskws89Io/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786334794676589-31872-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786334794676589-31872-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786334794676589-31872-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786334794676589-31872-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.32.1:0
--local_ip_for_outbound_sockets=127.31.32.1
--webserver_interface=127.31.32.1
--webserver_port=0
--tserver_master_addrs=127.31.32.62:39167
--builtin_ntp_servers=127.31.32.20:43129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260810 04:06:57.445734 4503 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:06:57.445920 4503 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:06:57.445941 4503 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:06:57.447058 4503 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:06:57.447094 4503 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:06:57.447118 4503 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.32.1
I20260810 04:06:57.448534 4503 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.32.20:43129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786334794676589-31872-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786334794676589-31872-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.32.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786334794676589-31872-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.32.1
--webserver_port=0
--tserver_master_addrs=127.31.32.62:39167
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4503
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.32.1
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786334794676589-31872-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:06:57.448778 4503 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:06:57.449075 4503 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:06:57.454149 4510 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:06:57.454202 4509 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:06:57.454350 4503 server_base.cc:1034] running on GCE node
W20260810 04:06:57.454504 4512 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:06:57.454721 4503 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:06:57.454968 4503 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:06:57.456173 4503 hybrid_clock.cc:648] HybridClock initialized: now 1786334817456145 us; error 44 us; skew 500 ppm
I20260810 04:06:57.457479 4503 webserver.cc:460] Webserver started at http://127.31.32.1:40377/ using document root <none> and password file <none>
I20260810 04:06:57.457731 4503 fs_manager.cc:359] Metadata directory not provided
I20260810 04:06:57.457799 4503 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:06:57.457967 4503 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:06:57.458977 4503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786334794676589-31872-0/minicluster-data/ts-0/data/instance:
uuid: "834a2435b28a489397122b9e010ac4f1"
format_stamp: "Formatted at 2026-08-10 04:06:57 on dist-test-slave-8w9v"
server_key: "3957800648a35f9908b5721c491aaec3"
server_key_iv: "c9ab1cff459978811fb6dc3c614fe7ec"
server_key_version: "encryptionkey@0"
I20260810 04:06:57.459339 4503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786334794676589-31872-0/minicluster-data/ts-0/wal/instance:
uuid: "834a2435b28a489397122b9e010ac4f1"
format_stamp: "Formatted at 2026-08-10 04:06:57 on dist-test-slave-8w9v"
server_key: "3957800648a35f9908b5721c491aaec3"
server_key_iv: "c9ab1cff459978811fb6dc3c614fe7ec"
server_key_version: "encryptionkey@0"
I20260810 04:06:57.460807 4503 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 04:06:57.461702 4518 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:06:57.461861 4503 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:06:57.461963 4503 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786334794676589-31872-0/minicluster-data/ts-0/data,/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786334794676589-31872-0/minicluster-data/ts-0/wal
uuid: "834a2435b28a489397122b9e010ac4f1"
format_stamp: "Formatted at 2026-08-10 04:06:57 on dist-test-slave-8w9v"
server_key: "3957800648a35f9908b5721c491aaec3"
server_key_iv: "c9ab1cff459978811fb6dc3c614fe7ec"
server_key_version: "encryptionkey@0"
I20260810 04:06:57.462034 4503 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786334794676589-31872-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786334794676589-31872-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786334794676589-31872-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:06:57.465348 4503 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:06:57.465611 4503 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:06:57.465739 4503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:06:57.465955 4503 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:06:57.466246 4503 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:06:57.466282 4503 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:06:57.466333 4503 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:06:57.466358 4503 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:06:57.473145 4503 rpc_server.cc:307] RPC server started. Bound to: 127.31.32.1:43947
I20260810 04:06:57.473187 4640 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.32.1:43947 every 8 connection(s)
I20260810 04:06:57.473480 4503 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786334794676589-31872-0/minicluster-data/ts-0/data/info.pb
I20260810 04:06:57.475296 31872 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskws89Io/build/release/bin/kudu as pid 4503
I20260810 04:06:57.475412 31872 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786334794676589-31872-0/minicluster-data/ts-0/wal/instance
I20260810 04:06:57.475606 31872 external_mini_cluster.cc:1428] Setting key 137daa2c628975b3229f5836633084e9
I20260810 04:06:57.476620 4500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:57.474222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.1:41747 (local address 127.31.32.62:39167)
0810 04:06:57.474285 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0810 04:06:57.474287 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:06:57.474510 (+ 223us) server_negotiation.cc:407] Connection header received
0810 04:06:57.474937 (+ 427us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:06:57.474938 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:06:57.474945 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:06:57.475029 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:06:57.476375 (+ 1346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:06:57.476376 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:06:57.476376 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:06:57.476380 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:06:57.476397 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:06:57.476401 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:06:57.476445 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0810 04:06:57.476533 (+ 88us) server_negotiation.cc:299] Negotiation successful
0810 04:06:57.476552 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260810 04:06:57.476742 31872 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskws89Io/build/release/bin/kudu
/tmp/dist-test-taskws89Io/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786334794676589-31872-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786334794676589-31872-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786334794676589-31872-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786334794676589-31872-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.32.2:0
--local_ip_for_outbound_sockets=127.31.32.2
--webserver_interface=127.31.32.2
--webserver_port=0
--tserver_master_addrs=127.31.32.62:39167
--builtin_ntp_servers=127.31.32.20:43129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260810 04:06:57.476806 4643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:57.474202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:39167 (local address 127.31.32.1:41747)
0810 04:06:57.474377 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0810 04:06:57.474416 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0810 04:06:57.474848 (+ 432us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:06:57.475077 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:06:57.475080 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:06:57.475088 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:06:57.476219 (+ 1131us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 04:06:57.476220 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:06:57.476255 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 04:06:57.476290 (+ 35us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:06:57.476454 (+ 164us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:06:57.476457 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:06:57.476459 (+ 2us) client_negotiation.cc:769] Sending connection context
0810 04:06:57.476537 (+ 78us) client_negotiation.cc:240] Negotiation successful
0810 04:06:57.476593 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":79,"threads_started":1}
I20260810 04:06:57.477408 4641 heartbeater.cc:347] Connected to a master server at 127.31.32.62:39167
I20260810 04:06:57.477500 4641 heartbeater.cc:464] Registering TS with master...
I20260810 04:06:57.477656 4641 heartbeater.cc:511] Master 127.31.32.62:39167 requested a full tablet report, sending...
I20260810 04:06:57.478194 4430 ts_manager.cc:194] Registered new tserver with Master: 834a2435b28a489397122b9e010ac4f1 (127.31.32.1:43947)
W20260810 04:06:57.557686 4644 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:06:57.557876 4644 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:06:57.557916 4644 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:06:57.559079 4644 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:06:57.559131 4644 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:06:57.559156 4644 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.32.2
I20260810 04:06:57.560417 4644 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.32.20:43129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786334794676589-31872-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786334794676589-31872-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.32.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786334794676589-31872-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.32.2
--webserver_port=0
--tserver_master_addrs=127.31.32.62:39167
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4644
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.32.2
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786334794676589-31872-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:06:57.560644 4644 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:06:57.560942 4644 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 04:06:57.566936 4644 server_base.cc:1034] running on GCE node
W20260810 04:06:57.566915 4650 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:06:57.566917 4651 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:06:57.566939 4653 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:06:57.567408 4644 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:06:57.567636 4644 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:06:57.568794 4644 hybrid_clock.cc:648] HybridClock initialized: now 1786334817568788 us; error 36 us; skew 500 ppm
I20260810 04:06:57.570037 4644 webserver.cc:460] Webserver started at http://127.31.32.2:41369/ using document root <none> and password file <none>
I20260810 04:06:57.570289 4644 fs_manager.cc:359] Metadata directory not provided
I20260810 04:06:57.570371 4644 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:06:57.570523 4644 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:06:57.571557 4644 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786334794676589-31872-0/minicluster-data/ts-1/data/instance:
uuid: "edd785117b884fc586831735e2474d30"
format_stamp: "Formatted at 2026-08-10 04:06:57 on dist-test-slave-8w9v"
server_key: "fed1c722107c521dacf26d4217ffe366"
server_key_iv: "028605072d2172cfea2fe94d2191c005"
server_key_version: "encryptionkey@0"
I20260810 04:06:57.571980 4644 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786334794676589-31872-0/minicluster-data/ts-1/wal/instance:
uuid: "edd785117b884fc586831735e2474d30"
format_stamp: "Formatted at 2026-08-10 04:06:57 on dist-test-slave-8w9v"
server_key: "fed1c722107c521dacf26d4217ffe366"
server_key_iv: "028605072d2172cfea2fe94d2191c005"
server_key_version: "encryptionkey@0"
I20260810 04:06:57.573365 4644 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260810 04:06:57.574090 4660 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:06:57.574270 4644 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 04:06:57.574368 4644 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786334794676589-31872-0/minicluster-data/ts-1/data,/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786334794676589-31872-0/minicluster-data/ts-1/wal
uuid: "edd785117b884fc586831735e2474d30"
format_stamp: "Formatted at 2026-08-10 04:06:57 on dist-test-slave-8w9v"
server_key: "fed1c722107c521dacf26d4217ffe366"
server_key_iv: "028605072d2172cfea2fe94d2191c005"
server_key_version: "encryptionkey@0"
I20260810 04:06:57.574450 4644 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786334794676589-31872-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786334794676589-31872-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786334794676589-31872-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:06:57.577857 4644 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:06:57.578121 4644 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:06:57.578262 4644 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:06:57.578467 4644 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:06:57.578764 4644 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:06:57.578816 4644 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:06:57.578862 4644 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:06:57.578890 4644 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:06:57.585067 4644 rpc_server.cc:307] RPC server started. Bound to: 127.31.32.2:43893
I20260810 04:06:57.585119 4779 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.32.2:43893 every 8 connection(s)
I20260810 04:06:57.585413 4644 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786334794676589-31872-0/minicluster-data/ts-1/data/info.pb
I20260810 04:06:57.588315 4500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:57.586142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.2:37497 (local address 127.31.32.62:39167)
0810 04:06:57.586210 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0810 04:06:57.586212 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:06:57.586408 (+ 196us) server_negotiation.cc:407] Connection header received
0810 04:06:57.586843 (+ 435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:06:57.586843 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:06:57.586848 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:06:57.586911 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:06:57.588067 (+ 1156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:06:57.588068 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:06:57.588068 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:06:57.588071 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:06:57.588083 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:06:57.588086 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:06:57.588185 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0810 04:06:57.588252 (+ 67us) server_negotiation.cc:299] Negotiation successful
0810 04:06:57.588264 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260810 04:06:57.588537 4782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:57.586105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:39167 (local address 127.31.32.2:37497)
0810 04:06:57.586283 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0810 04:06:57.586322 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0810 04:06:57.586759 (+ 437us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:06:57.586961 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:06:57.586964 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:06:57.586972 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:06:57.587940 (+ 968us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 04:06:57.587941 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:06:57.587968 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 04:06:57.587996 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:06:57.588187 (+ 191us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:06:57.588191 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:06:57.588193 (+ 2us) client_negotiation.cc:769] Sending connection context
0810 04:06:57.588260 (+ 67us) client_negotiation.cc:240] Negotiation successful
0810 04:06:57.588308 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":88,"threads_started":1}
I20260810 04:06:57.589046 4780 heartbeater.cc:347] Connected to a master server at 127.31.32.62:39167
I20260810 04:06:57.589136 4780 heartbeater.cc:464] Registering TS with master...
I20260810 04:06:57.589244 4780 heartbeater.cc:511] Master 127.31.32.62:39167 requested a full tablet report, sending...
I20260810 04:06:57.589637 4430 ts_manager.cc:194] Registered new tserver with Master: edd785117b884fc586831735e2474d30 (127.31.32.2:43893)
I20260810 04:06:57.592334 31872 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskws89Io/build/release/bin/kudu as pid 4644
I20260810 04:06:57.592438 31872 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786334794676589-31872-0/minicluster-data/ts-1/wal/instance
I20260810 04:06:57.592609 31872 external_mini_cluster.cc:1428] Setting key d4fbed083a56783786d847683dd5c94c
I20260810 04:06:57.593511 31872 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskws89Io/build/release/bin/kudu
/tmp/dist-test-taskws89Io/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786334794676589-31872-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786334794676589-31872-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786334794676589-31872-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786334794676589-31872-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.32.3:0
--local_ip_for_outbound_sockets=127.31.32.3
--webserver_interface=127.31.32.3
--webserver_port=0
--tserver_master_addrs=127.31.32.62:39167
--builtin_ntp_servers=127.31.32.20:43129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260810 04:06:57.672775 4783 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:06:57.672955 4783 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:06:57.672976 4783 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:06:57.674136 4783 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:06:57.674176 4783 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:06:57.674213 4783 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.32.3
I20260810 04:06:57.675577 4783 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.32.20:43129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786334794676589-31872-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786334794676589-31872-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.32.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786334794676589-31872-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.32.3
--webserver_port=0
--tserver_master_addrs=127.31.32.62:39167
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4783
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.32.3
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786334794676589-31872-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:06:57.675819 4783 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:06:57.676113 4783 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:06:57.680994 4791 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:06:57.681011 4789 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:06:57.681036 4788 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:06:57.681264 4783 server_base.cc:1034] running on GCE node
I20260810 04:06:57.681571 4783 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:06:57.681797 4783 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:06:57.682940 4783 hybrid_clock.cc:648] HybridClock initialized: now 1786334817682928 us; error 27 us; skew 500 ppm
I20260810 04:06:57.684279 4783 webserver.cc:460] Webserver started at http://127.31.32.3:33203/ using document root <none> and password file <none>
I20260810 04:06:57.684546 4783 fs_manager.cc:359] Metadata directory not provided
I20260810 04:06:57.684598 4783 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:06:57.684726 4783 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:06:57.685781 4783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786334794676589-31872-0/minicluster-data/ts-2/data/instance:
uuid: "5dc51c1960724e22aabcbc73a585b43c"
format_stamp: "Formatted at 2026-08-10 04:06:57 on dist-test-slave-8w9v"
server_key: "c83582e500f88d12bddc833af8195ecf"
server_key_iv: "3a7a236e99f957344b792edd057b003f"
server_key_version: "encryptionkey@0"
I20260810 04:06:57.686151 4783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786334794676589-31872-0/minicluster-data/ts-2/wal/instance:
uuid: "5dc51c1960724e22aabcbc73a585b43c"
format_stamp: "Formatted at 2026-08-10 04:06:57 on dist-test-slave-8w9v"
server_key: "c83582e500f88d12bddc833af8195ecf"
server_key_iv: "3a7a236e99f957344b792edd057b003f"
server_key_version: "encryptionkey@0"
I20260810 04:06:57.687578 4783 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 04:06:57.688375 4799 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:06:57.688553 4783 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 04:06:57.688647 4783 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786334794676589-31872-0/minicluster-data/ts-2/data,/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786334794676589-31872-0/minicluster-data/ts-2/wal
uuid: "5dc51c1960724e22aabcbc73a585b43c"
format_stamp: "Formatted at 2026-08-10 04:06:57 on dist-test-slave-8w9v"
server_key: "c83582e500f88d12bddc833af8195ecf"
server_key_iv: "3a7a236e99f957344b792edd057b003f"
server_key_version: "encryptionkey@0"
I20260810 04:06:57.688720 4783 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786334794676589-31872-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786334794676589-31872-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786334794676589-31872-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:06:57.691982 4783 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:06:57.692263 4783 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:06:57.692414 4783 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:06:57.692628 4783 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:06:57.692930 4783 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:06:57.692970 4783 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:06:57.693023 4783 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:06:57.693051 4783 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:06:57.698647 4783 rpc_server.cc:307] RPC server started. Bound to: 127.31.32.3:44741
I20260810 04:06:57.698684 4919 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.32.3:44741 every 8 connection(s)
I20260810 04:06:57.699007 4783 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786334794676589-31872-0/minicluster-data/ts-2/data/info.pb
I20260810 04:06:57.701942 4500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:57.699769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.3:38173 (local address 127.31.32.62:39167)
0810 04:06:57.699850 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0810 04:06:57.699852 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:06:57.700067 (+ 215us) server_negotiation.cc:407] Connection header received
0810 04:06:57.700558 (+ 491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:06:57.700559 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:06:57.700564 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:06:57.700622 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:06:57.701761 (+ 1139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:06:57.701762 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:06:57.701762 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:06:57.701765 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:06:57.701774 (+ 9us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:06:57.701777 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:06:57.701826 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0810 04:06:57.701882 (+ 56us) server_negotiation.cc:299] Negotiation successful
0810 04:06:57.701893 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 04:06:57.702113 4922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:57.699752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:39167 (local address 127.31.32.3:38173)
0810 04:06:57.699942 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0810 04:06:57.699980 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0810 04:06:57.700471 (+ 491us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:06:57.700688 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:06:57.700691 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:06:57.700707 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:06:57.701612 (+ 905us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 04:06:57.701613 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:06:57.701640 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 04:06:57.701665 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:06:57.701828 (+ 163us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:06:57.701830 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:06:57.701833 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 04:06:57.701885 (+ 52us) client_negotiation.cc:240] Negotiation successful
0810 04:06:57.701931 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":103,"threads_started":1}
I20260810 04:06:57.702639 4920 heartbeater.cc:347] Connected to a master server at 127.31.32.62:39167
I20260810 04:06:57.702723 4920 heartbeater.cc:464] Registering TS with master...
I20260810 04:06:57.702822 4920 heartbeater.cc:511] Master 127.31.32.62:39167 requested a full tablet report, sending...
I20260810 04:06:57.703150 4430 ts_manager.cc:194] Registered new tserver with Master: 5dc51c1960724e22aabcbc73a585b43c (127.31.32.3:44741)
I20260810 04:06:57.708758 31872 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskws89Io/build/release/bin/kudu as pid 4783
I20260810 04:06:57.708870 31872 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786334794676589-31872-0/minicluster-data/ts-2/wal/instance
I20260810 04:06:57.709061 31872 external_mini_cluster.cc:1428] Setting key e21fa8cf2ad2a73897f6a910d23374e5
I20260810 04:06:57.709815 31872 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 04:06:57.712170 4500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:57.711202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48422 (local address 127.31.32.62:39167)
0810 04:06:57.711252 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0810 04:06:57.711254 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:06:57.711390 (+ 136us) server_negotiation.cc:407] Connection header received
0810 04:06:57.711451 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:06:57.711451 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:06:57.711455 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:06:57.711495 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:06:57.711975 (+ 480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:06:57.711976 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:06:57.711976 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:06:57.711979 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:06:57.711992 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:06:57.711996 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:06:57.712037 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0810 04:06:57.712069 (+ 32us) server_negotiation.cc:299] Negotiation successful
0810 04:06:57.712080 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260810 04:06:57.713567 4430 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48422:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 04:06:57.715052 4430 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 04:06:57.718900 4931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:57.717156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.1:43947 (local address 127.0.0.1:47734)
0810 04:06:57.717335 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0810 04:06:57.717377 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 04:06:57.717431 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:06:57.717857 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:06:57.717864 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:06:57.717871 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:06:57.718352 (+ 481us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 04:06:57.718352 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:06:57.718368 (+ 16us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 04:06:57.718392 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:06:57.718736 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:06:57.718742 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:06:57.718745 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 04:06:57.718794 (+ 49us) client_negotiation.cc:240] Negotiation successful
0810 04:06:57.718804 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":55,"threads_started":1}
I20260810 04:06:57.719027 4933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:57.717448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47734 (local address 127.31.32.1:43947)
0810 04:06:57.717645 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0810 04:06:57.717651 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 04:06:57.717662 (+ 11us) server_negotiation.cc:407] Connection header received
0810 04:06:57.717712 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:06:57.717715 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:06:57.717724 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:06:57.718096 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:06:57.718593 (+ 497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:06:57.718596 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:06:57.718597 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:06:57.718600 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:06:57.718630 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:06:57.718645 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:06:57.718873 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0810 04:06:57.718916 (+ 43us) server_negotiation.cc:299] Negotiation successful
0810 04:06:57.718931 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":103,"threads_started":1}
I20260810 04:06:57.719056 4928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:57.717146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.2:43893 (local address 127.0.0.1:39936)
0810 04:06:57.717330 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0810 04:06:57.717383 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0810 04:06:57.717437 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:06:57.717967 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:06:57.717968 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:06:57.717971 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:06:57.718523 (+ 552us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 04:06:57.718524 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:06:57.718530 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 04:06:57.718541 (+ 11us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:06:57.718786 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:06:57.718787 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:06:57.718788 (+ 1us) client_negotiation.cc:769] Sending connection context
0810 04:06:57.718819 (+ 31us) client_negotiation.cc:240] Negotiation successful
0810 04:06:57.718833 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":97,"threads_started":1}
I20260810 04:06:57.719069 4930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:57.717160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.3:44741 (local address 127.0.0.1:43424)
0810 04:06:57.717790 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0810 04:06:57.717801 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 04:06:57.717839 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:06:57.718282 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:06:57.718283 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:06:57.718285 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:06:57.718678 (+ 393us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 04:06:57.718679 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:06:57.718684 (+ 5us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 04:06:57.718693 (+ 9us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:06:57.718929 (+ 236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:06:57.718929 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:06:57.718930 (+ 1us) client_negotiation.cc:769] Sending connection context
0810 04:06:57.718981 (+ 51us) client_negotiation.cc:240] Negotiation successful
0810 04:06:57.718992 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":613,"thread_start_us":107,"threads_started":1}
I20260810 04:06:57.719244 4934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:57.717624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43424 (local address 127.31.32.3:44741)
0810 04:06:57.717815 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0810 04:06:57.717819 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 04:06:57.717832 (+ 13us) server_negotiation.cc:407] Connection header received
0810 04:06:57.717895 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:06:57.717898 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:06:57.717908 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:06:57.717978 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:06:57.718775 (+ 797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:06:57.718782 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:06:57.718783 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:06:57.718787 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:06:57.718834 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:06:57.718855 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:06:57.718915 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0810 04:06:57.719162 (+ 247us) server_negotiation.cc:299] Negotiation successful
0810 04:06:57.719178 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":82,"threads_started":1}
I20260810 04:06:57.719380 4932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:57.717440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39936 (local address 127.31.32.2:43893)
0810 04:06:57.717617 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0810 04:06:57.717621 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 04:06:57.717632 (+ 11us) server_negotiation.cc:407] Connection header received
0810 04:06:57.717657 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:06:57.717659 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:06:57.717668 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:06:57.718177 (+ 509us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:06:57.718640 (+ 463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:06:57.718647 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:06:57.718647 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:06:57.718651 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:06:57.718686 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:06:57.718708 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:06:57.719245 (+ 537us) server_negotiation.cc:1035] Waiting for connection context
0810 04:06:57.719270 (+ 25us) server_negotiation.cc:299] Negotiation successful
0810 04:06:57.719288 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":86,"threads_started":1}
I20260810 04:06:57.719808 4851 tablet_service.cc:1467] Processing CreateTablet for tablet 65f565b7931e4245a5af29014c7a30e7 (DEFAULT_TABLE table=test-table [id=e0824ad9681b4d2fa05fc5f89f202185]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:06:57.719808 4571 tablet_service.cc:1467] Processing CreateTablet for tablet 65f565b7931e4245a5af29014c7a30e7 (DEFAULT_TABLE table=test-table [id=e0824ad9681b4d2fa05fc5f89f202185]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:06:57.720178 4851 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65f565b7931e4245a5af29014c7a30e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:06:57.720193 4571 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65f565b7931e4245a5af29014c7a30e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:06:57.720738 4711 tablet_service.cc:1467] Processing CreateTablet for tablet 65f565b7931e4245a5af29014c7a30e7 (DEFAULT_TABLE table=test-table [id=e0824ad9681b4d2fa05fc5f89f202185]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:06:57.721081 4711 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65f565b7931e4245a5af29014c7a30e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:06:57.723404 4935 tablet_bootstrap.cc:492] T 65f565b7931e4245a5af29014c7a30e7 P 5dc51c1960724e22aabcbc73a585b43c: Bootstrap starting.
I20260810 04:06:57.723655 4937 tablet_bootstrap.cc:492] T 65f565b7931e4245a5af29014c7a30e7 P edd785117b884fc586831735e2474d30: Bootstrap starting.
I20260810 04:06:57.724084 4935 tablet_bootstrap.cc:654] T 65f565b7931e4245a5af29014c7a30e7 P 5dc51c1960724e22aabcbc73a585b43c: Neither blocks nor log segments found. Creating new log.
I20260810 04:06:57.724321 4937 tablet_bootstrap.cc:654] T 65f565b7931e4245a5af29014c7a30e7 P edd785117b884fc586831735e2474d30: Neither blocks nor log segments found. Creating new log.
I20260810 04:06:57.724340 4936 tablet_bootstrap.cc:492] T 65f565b7931e4245a5af29014c7a30e7 P 834a2435b28a489397122b9e010ac4f1: Bootstrap starting.
I20260810 04:06:57.724483 4935 log.cc:824] T 65f565b7931e4245a5af29014c7a30e7 P 5dc51c1960724e22aabcbc73a585b43c: Log is configured to *not* fsync() on all Append() calls
I20260810 04:06:57.724663 4937 log.cc:824] T 65f565b7931e4245a5af29014c7a30e7 P edd785117b884fc586831735e2474d30: Log is configured to *not* fsync() on all Append() calls
I20260810 04:06:57.725072 4936 tablet_bootstrap.cc:654] T 65f565b7931e4245a5af29014c7a30e7 P 834a2435b28a489397122b9e010ac4f1: Neither blocks nor log segments found. Creating new log.
I20260810 04:06:57.725385 4935 tablet_bootstrap.cc:492] T 65f565b7931e4245a5af29014c7a30e7 P 5dc51c1960724e22aabcbc73a585b43c: No bootstrap required, opened a new log
I20260810 04:06:57.725394 4936 log.cc:824] T 65f565b7931e4245a5af29014c7a30e7 P 834a2435b28a489397122b9e010ac4f1: Log is configured to *not* fsync() on all Append() calls
I20260810 04:06:57.725502 4935 ts_tablet_manager.cc:1397] T 65f565b7931e4245a5af29014c7a30e7 P 5dc51c1960724e22aabcbc73a585b43c: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260810 04:06:57.726161 4936 tablet_bootstrap.cc:492] T 65f565b7931e4245a5af29014c7a30e7 P 834a2435b28a489397122b9e010ac4f1: No bootstrap required, opened a new log
I20260810 04:06:57.726243 4936 ts_tablet_manager.cc:1397] T 65f565b7931e4245a5af29014c7a30e7 P 834a2435b28a489397122b9e010ac4f1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 04:06:57.728320 4935 raft_consensus.cc:348] T 65f565b7931e4245a5af29014c7a30e7 P 5dc51c1960724e22aabcbc73a585b43c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "834a2435b28a489397122b9e010ac4f1" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 43947 } } peers { permanent_uuid: "edd785117b884fc586831735e2474d30" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 43893 } } peers { permanent_uuid: "5dc51c1960724e22aabcbc73a585b43c" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 44741 } }
I20260810 04:06:57.728523 4935 raft_consensus.cc:374] T 65f565b7931e4245a5af29014c7a30e7 P 5dc51c1960724e22aabcbc73a585b43c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:06:57.728569 4935 raft_consensus.cc:729] T 65f565b7931e4245a5af29014c7a30e7 P 5dc51c1960724e22aabcbc73a585b43c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dc51c1960724e22aabcbc73a585b43c, State: Initialized, Role: FOLLOWER
I20260810 04:06:57.728700 4935 consensus_queue.cc:260] T 65f565b7931e4245a5af29014c7a30e7 P 5dc51c1960724e22aabcbc73a585b43c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "834a2435b28a489397122b9e010ac4f1" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 43947 } } peers { permanent_uuid: "edd785117b884fc586831735e2474d30" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 43893 } } peers { permanent_uuid: "5dc51c1960724e22aabcbc73a585b43c" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 44741 } }
I20260810 04:06:57.728806 4937 tablet_bootstrap.cc:492] T 65f565b7931e4245a5af29014c7a30e7 P edd785117b884fc586831735e2474d30: No bootstrap required, opened a new log
I20260810 04:06:57.728870 4937 ts_tablet_manager.cc:1397] T 65f565b7931e4245a5af29014c7a30e7 P edd785117b884fc586831735e2474d30: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260810 04:06:57.728955 4935 ts_tablet_manager.cc:1428] T 65f565b7931e4245a5af29014c7a30e7 P 5dc51c1960724e22aabcbc73a585b43c: Time spent starting tablet: real 0.003s user 0.003s sys 0.001s
I20260810 04:06:57.729110 4920 heartbeater.cc:503] Master 127.31.32.62:39167 was elected leader, sending a full tablet report...
I20260810 04:06:57.729023 4936 raft_consensus.cc:348] T 65f565b7931e4245a5af29014c7a30e7 P 834a2435b28a489397122b9e010ac4f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "834a2435b28a489397122b9e010ac4f1" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 43947 } } peers { permanent_uuid: "edd785117b884fc586831735e2474d30" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 43893 } } peers { permanent_uuid: "5dc51c1960724e22aabcbc73a585b43c" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 44741 } }
I20260810 04:06:57.729175 4936 raft_consensus.cc:374] T 65f565b7931e4245a5af29014c7a30e7 P 834a2435b28a489397122b9e010ac4f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:06:57.729213 4936 raft_consensus.cc:729] T 65f565b7931e4245a5af29014c7a30e7 P 834a2435b28a489397122b9e010ac4f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 834a2435b28a489397122b9e010ac4f1, State: Initialized, Role: FOLLOWER
I20260810 04:06:57.729324 4936 consensus_queue.cc:260] T 65f565b7931e4245a5af29014c7a30e7 P 834a2435b28a489397122b9e010ac4f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "834a2435b28a489397122b9e010ac4f1" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 43947 } } peers { permanent_uuid: "edd785117b884fc586831735e2474d30" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 43893 } } peers { permanent_uuid: "5dc51c1960724e22aabcbc73a585b43c" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 44741 } }
I20260810 04:06:57.729564 4936 ts_tablet_manager.cc:1428] T 65f565b7931e4245a5af29014c7a30e7 P 834a2435b28a489397122b9e010ac4f1: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260810 04:06:57.729707 4641 heartbeater.cc:503] Master 127.31.32.62:39167 was elected leader, sending a full tablet report...
I20260810 04:06:57.731279 4937 raft_consensus.cc:348] T 65f565b7931e4245a5af29014c7a30e7 P edd785117b884fc586831735e2474d30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "834a2435b28a489397122b9e010ac4f1" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 43947 } } peers { permanent_uuid: "edd785117b884fc586831735e2474d30" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 43893 } } peers { permanent_uuid: "5dc51c1960724e22aabcbc73a585b43c" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 44741 } }
I20260810 04:06:57.731428 4937 raft_consensus.cc:374] T 65f565b7931e4245a5af29014c7a30e7 P edd785117b884fc586831735e2474d30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:06:57.731475 4937 raft_consensus.cc:729] T 65f565b7931e4245a5af29014c7a30e7 P edd785117b884fc586831735e2474d30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edd785117b884fc586831735e2474d30, State: Initialized, Role: FOLLOWER
I20260810 04:06:57.731585 4937 consensus_queue.cc:260] T 65f565b7931e4245a5af29014c7a30e7 P edd785117b884fc586831735e2474d30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "834a2435b28a489397122b9e010ac4f1" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 43947 } } peers { permanent_uuid: "edd785117b884fc586831735e2474d30" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 43893 } } peers { permanent_uuid: "5dc51c1960724e22aabcbc73a585b43c" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 44741 } }
I20260810 04:06:57.731828 4937 ts_tablet_manager.cc:1428] T 65f565b7931e4245a5af29014c7a30e7 P edd785117b884fc586831735e2474d30: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 04:06:57.731894 4780 heartbeater.cc:503] Master 127.31.32.62:39167 was elected leader, sending a full tablet report...
W20260810 04:06:57.835981 4781 tablet.cc:2369] T 65f565b7931e4245a5af29014c7a30e7 P edd785117b884fc586831735e2474d30: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 04:06:57.949576 4921 tablet.cc:2369] T 65f565b7931e4245a5af29014c7a30e7 P 5dc51c1960724e22aabcbc73a585b43c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 04:06:57.974200 4642 tablet.cc:2369] T 65f565b7931e4245a5af29014c7a30e7 P 834a2435b28a489397122b9e010ac4f1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 04:06:58.027671 4943 raft_consensus.cc:482] T 65f565b7931e4245a5af29014c7a30e7 P 5dc51c1960724e22aabcbc73a585b43c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 04:06:58.027806 4943 raft_consensus.cc:504] T 65f565b7931e4245a5af29014c7a30e7 P 5dc51c1960724e22aabcbc73a585b43c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "834a2435b28a489397122b9e010ac4f1" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 43947 } } peers { permanent_uuid: "edd785117b884fc586831735e2474d30" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 43893 } } peers { permanent_uuid: "5dc51c1960724e22aabcbc73a585b43c" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 44741 } }
I20260810 04:06:58.028285 4943 leader_election.cc:288] T 65f565b7931e4245a5af29014c7a30e7 P 5dc51c1960724e22aabcbc73a585b43c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 834a2435b28a489397122b9e010ac4f1 (127.31.32.1:43947), edd785117b884fc586831735e2474d30 (127.31.32.2:43893)
I20260810 04:06:58.029659 4922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:58.028493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.2:43893 (local address 127.31.32.3:52099)
0810 04:06:58.028557 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0810 04:06:58.028571 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 04:06:58.028612 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:06:58.028796 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:06:58.028797 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:06:58.028799 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:06:58.029300 (+ 501us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 04:06:58.029301 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:06:58.029309 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 04:06:58.029334 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:06:58.029525 (+ 191us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:06:58.029526 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:06:58.029526 (+ 0us) client_negotiation.cc:769] Sending connection context
0810 04:06:58.029577 (+ 51us) client_negotiation.cc:240] Negotiation successful
0810 04:06:58.029591 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260810 04:06:58.029659 4932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:58.028579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.3:52099 (local address 127.31.32.2:43893)
0810 04:06:58.028630 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0810 04:06:58.028632 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:06:58.028642 (+ 10us) server_negotiation.cc:407] Connection header received
0810 04:06:58.028675 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:06:58.028676 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:06:58.028682 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:06:58.028759 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:06:58.029440 (+ 681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:06:58.029441 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:06:58.029442 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:06:58.029445 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:06:58.029461 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:06:58.029465 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:06:58.029527 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0810 04:06:58.029576 (+ 49us) server_negotiation.cc:299] Negotiation successful
0810 04:06:58.029591 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260810 04:06:58.029876 4933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:58.028598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.3:33943 (local address 127.31.32.1:43947)
0810 04:06:58.028661 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0810 04:06:58.028663 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:06:58.028808 (+ 145us) server_negotiation.cc:407] Connection header received
0810 04:06:58.028850 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:06:58.028852 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:06:58.028859 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:06:58.028914 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:06:58.029596 (+ 682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:06:58.029598 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:06:58.029599 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:06:58.029603 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:06:58.029617 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:06:58.029622 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:06:58.029679 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0810 04:06:58.029766 (+ 87us) server_negotiation.cc:299] Negotiation successful
0810 04:06:58.029783 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260810 04:06:58.030061 4946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:58.028512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.1:43947 (local address 127.31.32.3:33943)
0810 04:06:58.028705 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0810 04:06:58.028715 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 04:06:58.028768 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:06:58.028933 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:06:58.028934 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:06:58.028936 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:06:58.029479 (+ 543us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 04:06:58.029480 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:06:58.029487 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 04:06:58.029506 (+ 19us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:06:58.029681 (+ 175us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:06:58.029682 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:06:58.029683 (+ 1us) client_negotiation.cc:769] Sending connection context
0810 04:06:58.029963 (+ 280us) client_negotiation.cc:240] Negotiation successful
0810 04:06:58.029977 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":103,"threads_started":1}
I20260810 04:06:58.030066 4732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65f565b7931e4245a5af29014c7a30e7" candidate_uuid: "5dc51c1960724e22aabcbc73a585b43c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edd785117b884fc586831735e2474d30" is_pre_election: true
I20260810 04:06:58.030237 4732 raft_consensus.cc:2454] T 65f565b7931e4245a5af29014c7a30e7 P edd785117b884fc586831735e2474d30 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5dc51c1960724e22aabcbc73a585b43c in term 0.
I20260810 04:06:58.030380 4591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65f565b7931e4245a5af29014c7a30e7" candidate_uuid: "5dc51c1960724e22aabcbc73a585b43c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "834a2435b28a489397122b9e010ac4f1" is_pre_election: true
I20260810 04:06:58.030463 4802 leader_election.cc:302] T 65f565b7931e4245a5af29014c7a30e7 P 5dc51c1960724e22aabcbc73a585b43c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dc51c1960724e22aabcbc73a585b43c, edd785117b884fc586831735e2474d30; no voters:
I20260810 04:06:58.030525 4591 raft_consensus.cc:2454] T 65f565b7931e4245a5af29014c7a30e7 P 834a2435b28a489397122b9e010ac4f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5dc51c1960724e22aabcbc73a585b43c in term 0.
I20260810 04:06:58.030615 4943 raft_consensus.cc:2781] T 65f565b7931e4245a5af29014c7a30e7 P 5dc51c1960724e22aabcbc73a585b43c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 04:06:58.030669 4943 raft_consensus.cc:482] T 65f565b7931e4245a5af29014c7a30e7 P 5dc51c1960724e22aabcbc73a585b43c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 04:06:58.030702 4943 raft_consensus.cc:3037] T 65f565b7931e4245a5af29014c7a30e7 P 5dc51c1960724e22aabcbc73a585b43c [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:06:58.031443 4943 raft_consensus.cc:504] T 65f565b7931e4245a5af29014c7a30e7 P 5dc51c1960724e22aabcbc73a585b43c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "834a2435b28a489397122b9e010ac4f1" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 43947 } } peers { permanent_uuid: "edd785117b884fc586831735e2474d30" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 43893 } } peers { permanent_uuid: "5dc51c1960724e22aabcbc73a585b43c" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 44741 } }
I20260810 04:06:58.031582 4943 leader_election.cc:288] T 65f565b7931e4245a5af29014c7a30e7 P 5dc51c1960724e22aabcbc73a585b43c [CANDIDATE]: Term 1 election: Requested vote from peers 834a2435b28a489397122b9e010ac4f1 (127.31.32.1:43947), edd785117b884fc586831735e2474d30 (127.31.32.2:43893)
I20260810 04:06:58.031725 4591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65f565b7931e4245a5af29014c7a30e7" candidate_uuid: "5dc51c1960724e22aabcbc73a585b43c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "834a2435b28a489397122b9e010ac4f1"
I20260810 04:06:58.031731 4732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65f565b7931e4245a5af29014c7a30e7" candidate_uuid: "5dc51c1960724e22aabcbc73a585b43c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edd785117b884fc586831735e2474d30"
I20260810 04:06:58.031803 4591 raft_consensus.cc:3037] T 65f565b7931e4245a5af29014c7a30e7 P 834a2435b28a489397122b9e010ac4f1 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:06:58.031827 4732 raft_consensus.cc:3037] T 65f565b7931e4245a5af29014c7a30e7 P edd785117b884fc586831735e2474d30 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:06:58.032675 4732 raft_consensus.cc:2454] T 65f565b7931e4245a5af29014c7a30e7 P edd785117b884fc586831735e2474d30 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dc51c1960724e22aabcbc73a585b43c in term 1.
I20260810 04:06:58.032685 4591 raft_consensus.cc:2454] T 65f565b7931e4245a5af29014c7a30e7 P 834a2435b28a489397122b9e010ac4f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dc51c1960724e22aabcbc73a585b43c in term 1.
I20260810 04:06:58.032904 4802 leader_election.cc:302] T 65f565b7931e4245a5af29014c7a30e7 P 5dc51c1960724e22aabcbc73a585b43c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dc51c1960724e22aabcbc73a585b43c, edd785117b884fc586831735e2474d30; no voters:
I20260810 04:06:58.033059 4943 raft_consensus.cc:2781] T 65f565b7931e4245a5af29014c7a30e7 P 5dc51c1960724e22aabcbc73a585b43c [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:06:58.033224 4943 raft_consensus.cc:686] T 65f565b7931e4245a5af29014c7a30e7 P 5dc51c1960724e22aabcbc73a585b43c [term 1 LEADER]: Becoming Leader. State: Replica: 5dc51c1960724e22aabcbc73a585b43c, State: Running, Role: LEADER
I20260810 04:06:58.033327 4943 consensus_queue.cc:237] T 65f565b7931e4245a5af29014c7a30e7 P 5dc51c1960724e22aabcbc73a585b43c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "834a2435b28a489397122b9e010ac4f1" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 43947 } } peers { permanent_uuid: "edd785117b884fc586831735e2474d30" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 43893 } } peers { permanent_uuid: "5dc51c1960724e22aabcbc73a585b43c" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 44741 } }
I20260810 04:06:58.034188 4431 catalog_manager.cc:5495] T 65f565b7931e4245a5af29014c7a30e7 P 5dc51c1960724e22aabcbc73a585b43c reported cstate change: term changed from 0 to 1, leader changed from <none> to 5dc51c1960724e22aabcbc73a585b43c (127.31.32.3). New cstate: current_term: 1 leader_uuid: "5dc51c1960724e22aabcbc73a585b43c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "834a2435b28a489397122b9e010ac4f1" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 43947 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "edd785117b884fc586831735e2474d30" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 43893 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5dc51c1960724e22aabcbc73a585b43c" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 44741 } health_report { overall_health: HEALTHY } } }
I20260810 04:06:58.075223 4934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:58.074198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43440 (local address 127.31.32.3:44741)
0810 04:06:58.074276 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0810 04:06:58.074278 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:06:58.074288 (+ 10us) server_negotiation.cc:407] Connection header received
0810 04:06:58.074340 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:06:58.074341 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:06:58.074355 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:06:58.074421 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:06:58.074976 (+ 555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:06:58.074977 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:06:58.074978 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:06:58.074982 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:06:58.074998 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:06:58.075003 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:06:58.075077 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0810 04:06:58.075125 (+ 48us) server_negotiation.cc:299] Negotiation successful
0810 04:06:58.075140 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260810 04:06:58.077236 4732 raft_consensus.cc:1261] T 65f565b7931e4245a5af29014c7a30e7 P edd785117b884fc586831735e2474d30 [term 1 FOLLOWER]: Refusing update from remote peer 5dc51c1960724e22aabcbc73a585b43c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:06:58.077330 4591 raft_consensus.cc:1261] T 65f565b7931e4245a5af29014c7a30e7 P 834a2435b28a489397122b9e010ac4f1 [term 1 FOLLOWER]: Refusing update from remote peer 5dc51c1960724e22aabcbc73a585b43c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:06:58.077589 4947 consensus_queue.cc:1035] T 65f565b7931e4245a5af29014c7a30e7 P 5dc51c1960724e22aabcbc73a585b43c [LEADER]: Connected to new peer: Peer: permanent_uuid: "edd785117b884fc586831735e2474d30" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 43893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:06:58.077747 4943 consensus_queue.cc:1035] T 65f565b7931e4245a5af29014c7a30e7 P 5dc51c1960724e22aabcbc73a585b43c [LEADER]: Connected to new peer: Peer: permanent_uuid: "834a2435b28a489397122b9e010ac4f1" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 43947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:06:58.079972 4950 mvcc.cc:205] Tried to move back new op lower bound from 7316827414841561088 to 7316827414665093120. Current Snapshot: MvccSnapshot[applied={T|T < 7316827414841561088}]
I20260810 04:06:58.080955 4952 mvcc.cc:205] Tried to move back new op lower bound from 7316827414841561088 to 7316827414665093120. Current Snapshot: MvccSnapshot[applied={T|T < 7316827414841561088}]
I20260810 04:06:58.082931 4430 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48422:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 04:06:58.083022 4430 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48422:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 04:06:58.083758 4430 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 255a721df0514556b482787491b7f228: Sending DeleteTablet for 3 replicas of tablet 65f565b7931e4245a5af29014c7a30e7
I20260810 04:06:58.084091 4571 tablet_service.cc:1514] Processing DeleteTablet for tablet 65f565b7931e4245a5af29014c7a30e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:06:58 UTC) from {username='slave'} at 127.0.0.1:47734
I20260810 04:06:58.084107 4711 tablet_service.cc:1514] Processing DeleteTablet for tablet 65f565b7931e4245a5af29014c7a30e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:06:58 UTC) from {username='slave'} at 127.0.0.1:39936
I20260810 04:06:58.084348 4851 tablet_service.cc:1514] Processing DeleteTablet for tablet 65f565b7931e4245a5af29014c7a30e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:06:58 UTC) from {username='slave'} at 127.0.0.1:43424
I20260810 04:06:58.084455 31872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskws89Io/build/release/bin/kudu with pid 4503
I20260810 04:06:58.089018 4962 tablet_replica.cc:330] T 65f565b7931e4245a5af29014c7a30e7 P edd785117b884fc586831735e2474d30: stopping tablet replica
I20260810 04:06:58.089164 4962 raft_consensus.cc:2229] T 65f565b7931e4245a5af29014c7a30e7 P edd785117b884fc586831735e2474d30 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 04:06:58.089293 4962 raft_consensus.cc:2258] T 65f565b7931e4245a5af29014c7a30e7 P edd785117b884fc586831735e2474d30 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:06:58.090135 4962 ts_tablet_manager.cc:1905] T 65f565b7931e4245a5af29014c7a30e7 P edd785117b884fc586831735e2474d30: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:06:58.092137 4962 ts_tablet_manager.cc:1918] T 65f565b7931e4245a5af29014c7a30e7 P edd785117b884fc586831735e2474d30: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 04:06:58.092202 4962 log.cc:1196] T 65f565b7931e4245a5af29014c7a30e7 P edd785117b884fc586831735e2474d30: Deleting WAL directory at /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786334794676589-31872-0/minicluster-data/ts-1/wal/wals/65f565b7931e4245a5af29014c7a30e7
I20260810 04:06:58.092463 4962 ts_tablet_manager.cc:1939] T 65f565b7931e4245a5af29014c7a30e7 P edd785117b884fc586831735e2474d30: Deleting consensus metadata
I20260810 04:06:58.096308 4963 tablet_replica.cc:330] T 65f565b7931e4245a5af29014c7a30e7 P 5dc51c1960724e22aabcbc73a585b43c: stopping tablet replica
I20260810 04:06:58.096431 4963 raft_consensus.cc:2229] T 65f565b7931e4245a5af29014c7a30e7 P 5dc51c1960724e22aabcbc73a585b43c [term 1 LEADER]: Raft consensus shutting down.
I20260810 04:06:58.096587 4963 raft_consensus.cc:2258] T 65f565b7931e4245a5af29014c7a30e7 P 5dc51c1960724e22aabcbc73a585b43c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:06:58.097211 4963 ts_tablet_manager.cc:1905] T 65f565b7931e4245a5af29014c7a30e7 P 5dc51c1960724e22aabcbc73a585b43c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:06:58.098130 4424 catalog_manager.cc:4841] TS edd785117b884fc586831735e2474d30 (127.31.32.2:43893): tablet 65f565b7931e4245a5af29014c7a30e7 (table test-table [id=e0824ad9681b4d2fa05fc5f89f202185]) successfully deleted
I20260810 04:06:58.099287 4963 ts_tablet_manager.cc:1918] T 65f565b7931e4245a5af29014c7a30e7 P 5dc51c1960724e22aabcbc73a585b43c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 04:06:58.099378 4963 log.cc:1196] T 65f565b7931e4245a5af29014c7a30e7 P 5dc51c1960724e22aabcbc73a585b43c: Deleting WAL directory at /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786334794676589-31872-0/minicluster-data/ts-2/wal/wals/65f565b7931e4245a5af29014c7a30e7
I20260810 04:06:58.099686 4963 ts_tablet_manager.cc:1939] T 65f565b7931e4245a5af29014c7a30e7 P 5dc51c1960724e22aabcbc73a585b43c: Deleting consensus metadata
I20260810 04:06:58.100193 4425 catalog_manager.cc:4841] TS 5dc51c1960724e22aabcbc73a585b43c (127.31.32.3:44741): tablet 65f565b7931e4245a5af29014c7a30e7 (table test-table [id=e0824ad9681b4d2fa05fc5f89f202185]) successfully deleted
W20260810 04:06:58.102277 4423 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.32.1:43947 (error 108)
I20260810 04:06:58.102671 31872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskws89Io/build/release/bin/kudu with pid 4644
I20260810 04:06:58.106021 4928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:58.102834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.1:43947 (local address 127.0.0.1:47742)
0810 04:06:58.105912 (+ 3078us) negotiation.cc:107] Waiting for socket to connect
0810 04:06:58.105973 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.1:43947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3051}
W20260810 04:06:58.113018 4423 catalog_manager.cc:4584] TS 834a2435b28a489397122b9e010ac4f1 (127.31.32.1:43947): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 65f565b7931e4245a5af29014c7a30e7: Network error: Client connection negotiation failed: client connection to 127.31.32.1:43947: connect: Connection refused (error 111)
I20260810 04:06:58.117646 31872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskws89Io/build/release/bin/kudu with pid 4783
I20260810 04:06:58.123907 31872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskws89Io/build/release/bin/kudu with pid 4407
2026-08-10T04:06:58Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (942 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260810 04:06:58.144575 31872 mini_oidc.cc:152] Starting JWKS server
I20260810 04:06:58.145819 31872 webserver.cc:460] Webserver started at https://127.0.0.1:36001/ using document root <none> and password file <none>
I20260810 04:06:58.145936 31872 mini_oidc.cc:189] Starting OIDC Discovery server
I20260810 04:06:58.146099 31872 webserver.cc:460] Webserver started at http://127.0.0.1:32953/ using document root <none> and password file <none>
2026-08-10T04:06:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T04:06:58Z Disabled control of system clock
I20260810 04:06:58.154328 31872 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskws89Io/build/release/bin/kudu
/tmp/dist-test-taskws89Io/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786334794676589-31872-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786334794676589-31872-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786334794676589-31872-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786334794676589-31872-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.32.62:46669
--webserver_interface=127.31.32.62
--webserver_port=0
--builtin_ntp_servers=127.31.32.20:45915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.32.62:46669
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36001/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786334794676589-31872-0/testchainca.cert with env {}
W20260810 04:06:58.234483 4970 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:06:58.234679 4970 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:06:58.234701 4970 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:06:58.235769 4970 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:06:58.235800 4970 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:06:58.235811 4970 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:06:58.235823 4970 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:06:58.235837 4970 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260810 04:06:58.235849 4970 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260810 04:06:58.235860 4970 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36001/jwks.json
I20260810 04:06:58.237458 4970 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.32.20:45915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786334794676589-31872-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786334794676589-31872-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.32.62:46669
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.32.62:46669
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36001/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786334794676589-31872-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.32.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786334794676589-31872-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4970
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786334794676589-31872-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:06:58.237645 4970 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:06:58.237802 4970 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:06:58.242735 4979 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:06:58.242884 4970 server_base.cc:1034] running on GCE node
W20260810 04:06:58.242780 4977 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:06:58.242730 4976 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:06:58.243301 4970 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:06:58.243597 4970 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:06:58.244760 4970 hybrid_clock.cc:648] HybridClock initialized: now 1786334818244734 us; error 38 us; skew 500 ppm
I20260810 04:06:58.245996 4970 webserver.cc:460] Webserver started at http://127.31.32.62:43949/ using document root <none> and password file <none>
I20260810 04:06:58.246230 4970 fs_manager.cc:359] Metadata directory not provided
I20260810 04:06:58.246300 4970 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:06:58.246466 4970 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:06:58.247601 4970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786334794676589-31872-0/minicluster-data/master-0/data/instance:
uuid: "d304ae299b8f4bf7a37160fb2cf62725"
format_stamp: "Formatted at 2026-08-10 04:06:58 on dist-test-slave-8w9v"
server_key: "2b8f6af6d291c4f80e62a572d5b5cd9e"
server_key_iv: "1395140707e4b41431088f2cf1022c76"
server_key_version: "encryptionkey@0"
I20260810 04:06:58.248018 4970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786334794676589-31872-0/minicluster-data/master-0/wal/instance:
uuid: "d304ae299b8f4bf7a37160fb2cf62725"
format_stamp: "Formatted at 2026-08-10 04:06:58 on dist-test-slave-8w9v"
server_key: "2b8f6af6d291c4f80e62a572d5b5cd9e"
server_key_iv: "1395140707e4b41431088f2cf1022c76"
server_key_version: "encryptionkey@0"
I20260810 04:06:58.249514 4970 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260810 04:06:58.250192 4985 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:06:58.250371 4970 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 04:06:58.250458 4970 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786334794676589-31872-0/minicluster-data/master-0/data,/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786334794676589-31872-0/minicluster-data/master-0/wal
uuid: "d304ae299b8f4bf7a37160fb2cf62725"
format_stamp: "Formatted at 2026-08-10 04:06:58 on dist-test-slave-8w9v"
server_key: "2b8f6af6d291c4f80e62a572d5b5cd9e"
server_key_iv: "1395140707e4b41431088f2cf1022c76"
server_key_version: "encryptionkey@0"
I20260810 04:06:58.250511 4970 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786334794676589-31872-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786334794676589-31872-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786334794676589-31872-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:06:58.262977 4970 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:06:58.263234 4970 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:06:58.263342 4970 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:06:58.267184 4970 rpc_server.cc:307] RPC server started. Bound to: 127.31.32.62:46669
I20260810 04:06:58.267246 5040 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.32.62:46669 every 8 connection(s)
I20260810 04:06:58.267553 4970 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786334794676589-31872-0/minicluster-data/master-0/data/info.pb
I20260810 04:06:58.268507 31872 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskws89Io/build/release/bin/kudu as pid 4970
I20260810 04:06:58.268643 31872 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786334794676589-31872-0/minicluster-data/master-0/wal/instance
I20260810 04:06:58.268828 5041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:06:58.268898 31872 external_mini_cluster.cc:1428] Setting key 01a540dcf8bbeed224488f58ff9fe7b4
I20260810 04:06:58.271575 5041 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d304ae299b8f4bf7a37160fb2cf62725: Bootstrap starting.
I20260810 04:06:58.272363 5041 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d304ae299b8f4bf7a37160fb2cf62725: Neither blocks nor log segments found. Creating new log.
I20260810 04:06:58.272658 5041 log.cc:824] T 00000000000000000000000000000000 P d304ae299b8f4bf7a37160fb2cf62725: Log is configured to *not* fsync() on all Append() calls
I20260810 04:06:58.273340 5041 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d304ae299b8f4bf7a37160fb2cf62725: No bootstrap required, opened a new log
I20260810 04:06:58.274354 5044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:58.269395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53886 (local address 127.31.32.62:46669)
0810 04:06:58.269637 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0810 04:06:58.269644 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 04:06:58.269668 (+ 24us) server_negotiation.cc:407] Connection header received
0810 04:06:58.270204 (+ 536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:06:58.270210 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:06:58.270270 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:06:58.270425 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:06:58.270649 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:58.271491 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:58.272363 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:58.272855 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:58.272986 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:06:58.273718 (+ 732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:06:58.273721 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:06:58.273722 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:06:58.273912 (+ 190us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:06:58.273945 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:06:58.273956 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:06:58.274033 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0810 04:06:58.274110 (+ 77us) server_negotiation.cc:299] Negotiation successful
0810 04:06:58.274189 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":95,"threads_started":1}
I20260810 04:06:58.275367 31872 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 04:06:58.275521 5041 raft_consensus.cc:348] T 00000000000000000000000000000000 P d304ae299b8f4bf7a37160fb2cf62725 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d304ae299b8f4bf7a37160fb2cf62725" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 46669 } }
I20260810 04:06:58.275658 5041 raft_consensus.cc:374] T 00000000000000000000000000000000 P d304ae299b8f4bf7a37160fb2cf62725 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:06:58.275681 5041 raft_consensus.cc:729] T 00000000000000000000000000000000 P d304ae299b8f4bf7a37160fb2cf62725 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d304ae299b8f4bf7a37160fb2cf62725, State: Initialized, Role: FOLLOWER
I20260810 04:06:58.275840 5041 consensus_queue.cc:260] T 00000000000000000000000000000000 P d304ae299b8f4bf7a37160fb2cf62725 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d304ae299b8f4bf7a37160fb2cf62725" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 46669 } }
I20260810 04:06:58.275900 5041 raft_consensus.cc:388] T 00000000000000000000000000000000 P d304ae299b8f4bf7a37160fb2cf62725 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 04:06:58.275921 5041 raft_consensus.cc:482] T 00000000000000000000000000000000 P d304ae299b8f4bf7a37160fb2cf62725 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 04:06:58.275987 5041 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d304ae299b8f4bf7a37160fb2cf62725 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:06:58.276686 5041 raft_consensus.cc:504] T 00000000000000000000000000000000 P d304ae299b8f4bf7a37160fb2cf62725 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d304ae299b8f4bf7a37160fb2cf62725" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 46669 } }
I20260810 04:06:58.276806 5041 leader_election.cc:302] T 00000000000000000000000000000000 P d304ae299b8f4bf7a37160fb2cf62725 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d304ae299b8f4bf7a37160fb2cf62725; no voters:
I20260810 04:06:58.276968 5041 leader_election.cc:288] T 00000000000000000000000000000000 P d304ae299b8f4bf7a37160fb2cf62725 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 04:06:58.277048 5046 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d304ae299b8f4bf7a37160fb2cf62725 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:06:58.277189 5046 raft_consensus.cc:686] T 00000000000000000000000000000000 P d304ae299b8f4bf7a37160fb2cf62725 [term 1 LEADER]: Becoming Leader. State: Replica: d304ae299b8f4bf7a37160fb2cf62725, State: Running, Role: LEADER
I20260810 04:06:58.277221 5041 sys_catalog.cc:564] T 00000000000000000000000000000000 P d304ae299b8f4bf7a37160fb2cf62725 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:06:58.277275 5046 consensus_queue.cc:237] T 00000000000000000000000000000000 P d304ae299b8f4bf7a37160fb2cf62725 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d304ae299b8f4bf7a37160fb2cf62725" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 46669 } }
I20260810 04:06:58.277661 5047 sys_catalog.cc:455] T 00000000000000000000000000000000 P d304ae299b8f4bf7a37160fb2cf62725 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d304ae299b8f4bf7a37160fb2cf62725" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d304ae299b8f4bf7a37160fb2cf62725" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 46669 } } }
I20260810 04:06:58.277700 5048 sys_catalog.cc:455] T 00000000000000000000000000000000 P d304ae299b8f4bf7a37160fb2cf62725 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d304ae299b8f4bf7a37160fb2cf62725. Latest consensus state: current_term: 1 leader_uuid: "d304ae299b8f4bf7a37160fb2cf62725" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d304ae299b8f4bf7a37160fb2cf62725" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 46669 } } }
I20260810 04:06:58.277760 5047 sys_catalog.cc:458] T 00000000000000000000000000000000 P d304ae299b8f4bf7a37160fb2cf62725 [sys.catalog]: This master's current role is: LEADER
I20260810 04:06:58.277783 5048 sys_catalog.cc:458] T 00000000000000000000000000000000 P d304ae299b8f4bf7a37160fb2cf62725 [sys.catalog]: This master's current role is: LEADER
I20260810 04:06:58.278082 5055 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 04:06:58.278632 5055 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 04:06:58.280073 5055 catalog_manager.cc:1347] Generated new cluster ID: 10bb892f916f4ffaabcfdeabaef5a634
I20260810 04:06:58.280153 5055 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 04:06:58.304888 5055 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 04:06:58.305418 5055 catalog_manager.cc:1504] Loading token signing keys...
I20260810 04:06:58.318894 5055 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d304ae299b8f4bf7a37160fb2cf62725: Generated new TSK 0
I20260810 04:06:58.319092 5055 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260810 04:06:58.350378 5072 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260810 04:06:58.350486 5072 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260810 04:06:58.350801 5044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:58.335224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53900 (local address 127.31.32.62:46669)
0810 04:06:58.335277 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0810 04:06:58.335279 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:06:58.335416 (+ 137us) server_negotiation.cc:407] Connection header received
0810 04:06:58.335478 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:06:58.335479 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:06:58.335493 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:06:58.335573 (+ 80us) server_negotiation.cc:226] Negotiated authn=JWT
0810 04:06:58.335721 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:58.336513 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:58.337334 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:58.337630 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:58.337700 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:06:58.337891 (+ 191us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0810 04:06:58.350546 (+ 12655us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36001/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0810 04:06:58.350735 (+ 189us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:53900: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36001/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":31}
W20260810 04:06:58.350884 5044 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53900: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36001/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260810 04:06:58.351173 5071 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:06:58.335156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:46669 (local address 127.0.0.1:53900)
0810 04:06:58.335340 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0810 04:06:58.335350 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 04:06:58.335407 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:06:58.335587 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:06:58.335587 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:06:58.335589 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0810 04:06:58.335650 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:06:58.335652 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:06:58.336588 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:06:58.336589 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:06:58.337253 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:06:58.337257 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:06:58.337706 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:06:58.337707 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:06:58.337749 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:06:58.337767 (+ 18us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0810 04:06:58.350705 (+ 12938us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36001/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0810 04:06:58.350791 (+ 86us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.31.32.62:46669: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36001/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":93,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.31.32.62:46669: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36001/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260810 04:06:58.351660 31872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskws89Io/build/release/bin/kudu with pid 4970
2026-08-10T04:06:58Z chronyd exiting
I20260810 04:06:58.390723 31872 test_util.cc:175] -----------------------------------------------
I20260810 04:06:58.390822 31872 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786334794676589-31872-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (247 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260810 04:06:58.391713 31872 mini_oidc.cc:152] Starting JWKS server
I20260810 04:06:58.392494 31872 webserver.cc:460] Webserver started at https://127.0.0.1:43167/ using document root <none> and password file <none>
I20260810 04:06:58.392566 31872 mini_oidc.cc:189] Starting OIDC Discovery server
I20260810 04:06:58.392741 31872 webserver.cc:460] Webserver started at http://127.0.0.1:43023/ using document root <none> and password file <none>
2026-08-10T04:06:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T04:06:58Z Disabled control of system clock
I20260810 04:06:58.401489 31872 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskws89Io/build/release/bin/kudu
/tmp/dist-test-taskws89Io/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786334794676589-31872-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786334794676589-31872-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786334794676589-31872-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786334794676589-31872-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.32.62:36419
--webserver_interface=127.31.32.62
--webserver_port=0
--builtin_ntp_servers=127.31.32.20:46133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.32.62:36419
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43167/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786334794676589-31872-0/testchainca.cert with env {}
W20260810 04:06:58.481894 5078 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:06:58.482091 5078 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:06:58.482115 5078 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:06:58.483201 5078 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:06:58.483234 5078 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:06:58.483246 5078 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:06:58.483258 5078 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:06:58.483270 5078 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260810 04:06:58.483283 5078 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260810 04:06:58.483294 5078 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:43167/jwks.json
I20260810 04:06:58.484841 5078 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.32.20:46133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786334794676589-31872-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786334794676589-31872-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.32.62:36419
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.32.62:36419
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43167/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786334794676589-31872-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.32.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786334794676589-31872-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5078
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786334794676589-31872-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:06:58.485033 5078 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:06:58.485255 5078 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:06:58.490072 5083 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:06:58.490157 5087 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:06:58.490099 5085 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:06:58.490296 5078 server_base.cc:1034] running on GCE node
I20260810 04:06:58.490517 5078 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:06:58.490804 5078 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:06:58.491950 5078 hybrid_clock.cc:648] HybridClock initialized: now 1786334818491927 us; error 49 us; skew 500 ppm
I20260810 04:06:58.493212 5078 webserver.cc:460] Webserver started at http://127.31.32.62:33577/ using document root <none> and password file <none>
I20260810 04:06:58.493471 5078 fs_manager.cc:359] Metadata directory not provided
I20260810 04:06:58.493520 5078 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:06:58.493608 5078 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:06:58.494637 5078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786334794676589-31872-0/minicluster-data/master-0/data/instance:
uuid: "aa72166151d047a8afabc85a8d8a1d09"
format_stamp: "Formatted at 2026-08-10 04:06:58 on dist-test-slave-8w9v"
server_key: "a7cc80d14d369127bc623120a71612e6"
server_key_iv: "7d4e108fe029f57567940c59ec24f381"
server_key_version: "encryptionkey@0"
I20260810 04:06:58.495040 5078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786334794676589-31872-0/minicluster-data/master-0/wal/instance:
uuid: "aa72166151d047a8afabc85a8d8a1d09"
format_stamp: "Formatted at 2026-08-10 04:06:58 on dist-test-slave-8w9v"
server_key: "a7cc80d14d369127bc623120a71612e6"
server_key_iv: "7d4e108fe029f57567940c59ec24f381"
server_key_version: "encryptionkey@0"
I20260810 04:06:58.496610 5078 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 04:06:58.497289 5093 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:06:58.497481 5078 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 04:06:58.497546 5078 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786334794676589-31872-0/minicluster-data/master-0/data,/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786334794676589-31872-0/minicluster-data/master-0/wal
uuid: "aa72166151d047a8afabc85a8d8a1d09"
format_stamp: "Formatted at 2026-08-10 04:06:58 on dist-test-slave-8w9v"
server_key: "a7cc80d14d369127bc623120a71612e6"
server_key_iv: "7d4e108fe029f57567940c59ec24f381"
server_key_version: "encryptionkey@0"
I20260810 04:06:58.497635 5078 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786334794676589-31872-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786334794676589-31872-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786334794676589-31872-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:06:58.517475 5078 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:06:58.517761 5078 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:06:58.517917 5078 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:06:58.522006 5078 rpc_server.cc:307] RPC server started. Bound to: 127.31.32.62:36419
I20260810 04:06:58.522043 5150 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.32.62:36419 every 8 connection(s)
I20260810 04:06:58.522378 5078 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786334794676589-31872-0/minicluster-data/master-0/data/info.pb
I20260810 04:06:58.523021 5152 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:06:58.525629 5152 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa72166151d047a8afabc85a8d8a1d09: Bootstrap starting.
I20260810 04:06:58.525741 31872 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskws89Io/build/release/bin/kudu as pid 5078
I20260810 04:06:58.525859 31872 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786334794676589-31872-0/minicluster-data/master-0/wal/instance
I20260810 04:06:58.526042 31872 external_mini_cluster.cc:1428] Setting key 8de6aafb671cbb0d96481b0a8d3c38cc
I20260810 04:06:58.526409 5152 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aa72166151d047a8afabc85a8d8a1d09: Neither blocks nor log segments found. Creating new log.
I20260810 04:06:58.526773 5152 log.cc:824] T 00000000000000000000000000000000 P aa72166151d047a8afabc85a8d8a1d09: Log is configured to *not* fsync() on all Append() calls
I20260810 04:06:58.527500 5152 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa72166151d047a8afabc85a8d8a1d09: No bootstrap required, opened a new log
I20260810 04:06:58.529683 5152 raft_consensus.cc:348] T 00000000000000000000000000000000 P aa72166151d047a8afabc85a8d8a1d09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa72166151d047a8afabc85a8d8a1d09" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 36419 } }
I20260810 04:06:58.529847 5152 raft_consensus.cc:374] T 00000000000000000000000000000000 P aa72166151d047a8afabc85a8d8a1d09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:06:58.529886 5152 raft_consensus.cc:729] T 00000000000000000000000000000000 P aa72166151d047a8afabc85a8d8a1d09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa72166151d047a8afabc85a8d8a1d09, State: Initialized, Role: FOLLOWER
I20260810 04:06:58.530031 5152 consensus_queue.cc:260] T 00000000000000000000000000000000 P aa72166151d047a8afabc85a8d8a1d09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa72166151d047a8afabc85a8d8a1d09" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 36419 } }
I20260810 04:06:58.530099 5152 raft_consensus.cc:388] T 00000000000000000000000000000000 P aa72166151d047a8afabc85a8d8a1d09 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 04:06:58.530119 5152 raft_consensus.cc:482] T 00000000000000000000000000000000 P aa72166151d047a8afabc85a8d8a1d09 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 04:06:58.530189 5152 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aa72166151d047a8afabc85a8d8a1d09 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:06:58.530826 5152 raft_consensus.cc:504] T 00000000000000000000000000000000 P aa72166151d047a8afabc85a8d8a1d09 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa72166151d047a8afabc85a8d8a1d09" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 36419 } }
I20260810 04:06:58.530939 5152 leader_election.cc:302] T 00000000000000000000000000000000 P aa72166151d047a8afabc85a8d8a1d09 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aa72166151d047a8afabc85a8d8a1d09; no voters:
I20260810 04:06:58.531116 5152 leader_election.cc:288] T 00000000000000000000000000000000 P aa72166151d047a8afabc85a8d8a1d09 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 04:06:58.531224 5157 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aa72166151d047a8afabc85a8d8a1d09 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:06:58.531370 5157 raft_consensus.cc:686] T 00000000000000000000000000000000 P aa72166151d047a8afabc85a8d8a1d09 [term 1 LEADER]: Becoming Leader. State: Replica: aa72166151d047a8afabc85a8d8a1d09, State: Running, Role: LEADER
I20260810 04:06:58.531379 5152 sys_catalog.cc:564] T 00000000000000000000000000000000 P aa72166151d047a8afabc85a8d8a1d09 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:06:58.531491 5157 consensus_queue.cc:237] T 00000000000000000000000000000000 P aa72166151d047a8afabc85a8d8a1d09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa72166151d047a8afabc85a8d8a1d09" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 36419 } }
I20260810 04:06:58.531827 5157 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa72166151d047a8afabc85a8d8a1d09 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aa72166151d047a8afabc85a8d8a1d09. Latest consensus state: current_term: 1 leader_uuid: "aa72166151d047a8afabc85a8d8a1d09" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa72166151d047a8afabc85a8d8a1d09" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 36419 } } }
I20260810 04:06:58.531904 5157 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa72166151d047a8afabc85a8d8a1d09 [sys.catalog]: This master's current role is: LEADER
I20260810 04:06:58.532055 5158 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa72166151d047a8afabc85a8d8a1d09 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aa72166151d047a8afabc85a8d8a1d09" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa72166151d047a8afabc85a8d8a1d09" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 36419 } } }
I20260810 04:06:58.532531 5158 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa72166151d047a8afabc85a8d8a1d09 [sys.catalog]: This master's current role is: LEADER
I20260810 04:06:58.532497 5164 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 04:06:58.533140 5164 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 04:06:58.532516 5155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:58.526548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49342 (local address 127.31.32.62:36419)
0810 04:06:58.526734 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0810 04:06:58.526741 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 04:06:58.526766 (+ 25us) server_negotiation.cc:407] Connection header received
0810 04:06:58.527373 (+ 607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:06:58.527381 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:06:58.527438 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:06:58.527552 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:06:58.527785 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:58.528732 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:58.529985 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:58.530495 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:58.530586 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:06:58.531324 (+ 738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:06:58.531327 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:06:58.531328 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:06:58.531523 (+ 195us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:06:58.531569 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:06:58.531588 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:06:58.531661 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0810 04:06:58.532201 (+ 540us) server_negotiation.cc:299] Negotiation successful
0810 04:06:58.532302 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":88,"threads_started":1}
I20260810 04:06:58.534639 31872 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 04:06:58.534653 5164 catalog_manager.cc:1347] Generated new cluster ID: 71fb323b9ed943cb9a00a89e8e83b8cf
I20260810 04:06:58.534693 5164 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 04:06:58.557377 5164 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 04:06:58.558000 5164 catalog_manager.cc:1504] Loading token signing keys...
I20260810 04:06:58.574558 5164 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aa72166151d047a8afabc85a8d8a1d09: Generated new TSK 0
I20260810 04:06:58.574733 5164 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260810 04:06:58.608804 5184 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260810 04:06:58.608909 5184 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260810 04:06:58.609179 5155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:58.588556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49350 (local address 127.31.32.62:36419)
0810 04:06:58.588618 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0810 04:06:58.588620 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:06:58.588724 (+ 104us) server_negotiation.cc:407] Connection header received
0810 04:06:58.588801 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:06:58.588802 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:06:58.588813 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:06:58.588909 (+ 96us) server_negotiation.cc:226] Negotiated authn=JWT
0810 04:06:58.589035 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:58.589504 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:58.590275 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:58.590478 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:58.590553 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:06:58.590739 (+ 186us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0810 04:06:58.608941 (+ 18202us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43167/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0810 04:06:58.609121 (+ 180us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:49350: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43167/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":38}
W20260810 04:06:58.609259 5155 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:49350: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43167/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260810 04:06:58.609289 5183 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:06:58.588508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:36419 (local address 127.0.0.1:49350)
0810 04:06:58.588656 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0810 04:06:58.588665 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 04:06:58.588728 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:06:58.588911 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:06:58.588912 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:06:58.588913 (+ 1us) client_negotiation.cc:189] Negotiated authn=JWT
0810 04:06:58.588971 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:06:58.588974 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:06:58.589590 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:06:58.589591 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:06:58.590189 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:06:58.590191 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:06:58.590554 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:06:58.590555 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:06:58.590633 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:06:58.590645 (+ 12us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0810 04:06:58.609076 (+ 18431us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43167/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0810 04:06:58.609195 (+ 119us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.31.32.62:36419: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43167/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":77,"threads_started":1}
I20260810 04:06:58.609874 31872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskws89Io/build/release/bin/kudu with pid 5078
2026-08-10T04:06:58Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (258 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260810 04:06:58.650671 31872 mini_oidc.cc:152] Starting JWKS server
I20260810 04:06:58.651340 31872 webserver.cc:460] Webserver started at https://127.0.0.1:38203/ using document root <none> and password file <none>
I20260810 04:06:58.651408 31872 mini_oidc.cc:189] Starting OIDC Discovery server
I20260810 04:06:58.651573 31872 webserver.cc:460] Webserver started at http://127.0.0.1:35697/ using document root <none> and password file <none>
2026-08-10T04:06:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T04:06:58Z Disabled control of system clock
I20260810 04:06:58.660159 31872 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskws89Io/build/release/bin/kudu
/tmp/dist-test-taskws89Io/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786334794676589-31872-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786334794676589-31872-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786334794676589-31872-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786334794676589-31872-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.32.62:45351
--webserver_interface=127.31.32.62
--webserver_port=0
--builtin_ntp_servers=127.31.32.20:41443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.32.62:45351
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38203/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260810 04:06:58.739583 5190 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:06:58.739825 5190 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:06:58.739862 5190 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:06:58.741106 5190 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:06:58.741174 5190 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:06:58.741199 5190 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:06:58.741231 5190 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:06:58.741254 5190 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260810 04:06:58.741283 5190 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260810 04:06:58.741312 5190 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:38203/jwks.json
I20260810 04:06:58.742658 5190 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.32.20:41443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786334794676589-31872-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786334794676589-31872-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.32.62:45351
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.32.62:45351
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38203/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786334794676589-31872-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.32.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5190
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786334794676589-31872-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:06:58.742928 5190 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:06:58.743202 5190 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:06:58.748018 5197 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:06:58.747997 5196 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:06:58.748008 5200 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:06:58.748148 5190 server_base.cc:1034] running on GCE node
I20260810 04:06:58.748631 5190 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:06:58.748910 5190 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:06:58.750073 5190 hybrid_clock.cc:648] HybridClock initialized: now 1786334818750061 us; error 44 us; skew 500 ppm
I20260810 04:06:58.751350 5190 webserver.cc:460] Webserver started at http://127.31.32.62:42363/ using document root <none> and password file <none>
I20260810 04:06:58.751610 5190 fs_manager.cc:359] Metadata directory not provided
I20260810 04:06:58.751684 5190 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:06:58.751838 5190 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:06:58.752944 5190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786334794676589-31872-0/minicluster-data/master-0/data/instance:
uuid: "af57b2dc49d84cc7a0fe7fd18be133a8"
format_stamp: "Formatted at 2026-08-10 04:06:58 on dist-test-slave-8w9v"
server_key: "715bd4b0732cab5b3bfe61a971351472"
server_key_iv: "ca53dd582c548cc5799a511832bc4874"
server_key_version: "encryptionkey@0"
I20260810 04:06:58.753474 5190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786334794676589-31872-0/minicluster-data/master-0/wal/instance:
uuid: "af57b2dc49d84cc7a0fe7fd18be133a8"
format_stamp: "Formatted at 2026-08-10 04:06:58 on dist-test-slave-8w9v"
server_key: "715bd4b0732cab5b3bfe61a971351472"
server_key_iv: "ca53dd582c548cc5799a511832bc4874"
server_key_version: "encryptionkey@0"
I20260810 04:06:58.755117 5190 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 04:06:58.755824 5206 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:06:58.755991 5190 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260810 04:06:58.756098 5190 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786334794676589-31872-0/minicluster-data/master-0/data,/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786334794676589-31872-0/minicluster-data/master-0/wal
uuid: "af57b2dc49d84cc7a0fe7fd18be133a8"
format_stamp: "Formatted at 2026-08-10 04:06:58 on dist-test-slave-8w9v"
server_key: "715bd4b0732cab5b3bfe61a971351472"
server_key_iv: "ca53dd582c548cc5799a511832bc4874"
server_key_version: "encryptionkey@0"
I20260810 04:06:58.756196 5190 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786334794676589-31872-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786334794676589-31872-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786334794676589-31872-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:06:58.768420 5190 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:06:58.768695 5190 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:06:58.768792 5190 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:06:58.772673 5190 rpc_server.cc:307] RPC server started. Bound to: 127.31.32.62:45351
I20260810 04:06:58.772706 5260 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.32.62:45351 every 8 connection(s)
I20260810 04:06:58.773023 5190 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786334794676589-31872-0/minicluster-data/master-0/data/info.pb
I20260810 04:06:58.773672 5261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:06:58.774041 31872 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskws89Io/build/release/bin/kudu as pid 5190
I20260810 04:06:58.774114 31872 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786334794676589-31872-0/minicluster-data/master-0/wal/instance
I20260810 04:06:58.774348 31872 external_mini_cluster.cc:1428] Setting key 5b71fe9a5906817111d44b835b1f3e58
I20260810 04:06:58.777535 5261 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af57b2dc49d84cc7a0fe7fd18be133a8: Bootstrap starting.
I20260810 04:06:58.778322 5261 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P af57b2dc49d84cc7a0fe7fd18be133a8: Neither blocks nor log segments found. Creating new log.
I20260810 04:06:58.778625 5261 log.cc:824] T 00000000000000000000000000000000 P af57b2dc49d84cc7a0fe7fd18be133a8: Log is configured to *not* fsync() on all Append() calls
I20260810 04:06:58.779479 5261 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af57b2dc49d84cc7a0fe7fd18be133a8: No bootstrap required, opened a new log
I20260810 04:06:58.780529 5264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:58.774855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54746 (local address 127.31.32.62:45351)
0810 04:06:58.775073 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0810 04:06:58.775082 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 04:06:58.775116 (+ 34us) server_negotiation.cc:407] Connection header received
0810 04:06:58.775663 (+ 547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:06:58.775673 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:06:58.775753 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:06:58.775901 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:06:58.776106 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:58.777506 (+ 1400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:58.778311 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:58.778915 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:58.779007 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:06:58.779761 (+ 754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:06:58.779766 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:06:58.779766 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:06:58.779961 (+ 195us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:06:58.780003 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:06:58.780017 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:06:58.780084 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0810 04:06:58.780194 (+ 110us) server_negotiation.cc:299] Negotiation successful
0810 04:06:58.780272 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":90,"threads_started":1}
I20260810 04:06:58.781665 31872 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 04:06:58.782022 5261 raft_consensus.cc:348] T 00000000000000000000000000000000 P af57b2dc49d84cc7a0fe7fd18be133a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af57b2dc49d84cc7a0fe7fd18be133a8" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 45351 } }
I20260810 04:06:58.782152 5261 raft_consensus.cc:374] T 00000000000000000000000000000000 P af57b2dc49d84cc7a0fe7fd18be133a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:06:58.782176 5261 raft_consensus.cc:729] T 00000000000000000000000000000000 P af57b2dc49d84cc7a0fe7fd18be133a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af57b2dc49d84cc7a0fe7fd18be133a8, State: Initialized, Role: FOLLOWER
I20260810 04:06:58.782316 5261 consensus_queue.cc:260] T 00000000000000000000000000000000 P af57b2dc49d84cc7a0fe7fd18be133a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af57b2dc49d84cc7a0fe7fd18be133a8" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 45351 } }
I20260810 04:06:58.782428 5261 raft_consensus.cc:388] T 00000000000000000000000000000000 P af57b2dc49d84cc7a0fe7fd18be133a8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 04:06:58.782476 5261 raft_consensus.cc:482] T 00000000000000000000000000000000 P af57b2dc49d84cc7a0fe7fd18be133a8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 04:06:58.782526 5261 raft_consensus.cc:3037] T 00000000000000000000000000000000 P af57b2dc49d84cc7a0fe7fd18be133a8 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:06:58.783061 5261 raft_consensus.cc:504] T 00000000000000000000000000000000 P af57b2dc49d84cc7a0fe7fd18be133a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af57b2dc49d84cc7a0fe7fd18be133a8" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 45351 } }
I20260810 04:06:58.783149 5261 leader_election.cc:302] T 00000000000000000000000000000000 P af57b2dc49d84cc7a0fe7fd18be133a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: af57b2dc49d84cc7a0fe7fd18be133a8; no voters:
I20260810 04:06:58.783272 5261 leader_election.cc:288] T 00000000000000000000000000000000 P af57b2dc49d84cc7a0fe7fd18be133a8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 04:06:58.783324 5266 raft_consensus.cc:2781] T 00000000000000000000000000000000 P af57b2dc49d84cc7a0fe7fd18be133a8 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:06:58.783514 5266 raft_consensus.cc:686] T 00000000000000000000000000000000 P af57b2dc49d84cc7a0fe7fd18be133a8 [term 1 LEADER]: Becoming Leader. State: Replica: af57b2dc49d84cc7a0fe7fd18be133a8, State: Running, Role: LEADER
I20260810 04:06:58.783522 5261 sys_catalog.cc:564] T 00000000000000000000000000000000 P af57b2dc49d84cc7a0fe7fd18be133a8 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:06:58.783610 5266 consensus_queue.cc:237] T 00000000000000000000000000000000 P af57b2dc49d84cc7a0fe7fd18be133a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af57b2dc49d84cc7a0fe7fd18be133a8" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 45351 } }
I20260810 04:06:58.784024 5268 sys_catalog.cc:455] T 00000000000000000000000000000000 P af57b2dc49d84cc7a0fe7fd18be133a8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader af57b2dc49d84cc7a0fe7fd18be133a8. Latest consensus state: current_term: 1 leader_uuid: "af57b2dc49d84cc7a0fe7fd18be133a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af57b2dc49d84cc7a0fe7fd18be133a8" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 45351 } } }
I20260810 04:06:58.784112 5268 sys_catalog.cc:458] T 00000000000000000000000000000000 P af57b2dc49d84cc7a0fe7fd18be133a8 [sys.catalog]: This master's current role is: LEADER
I20260810 04:06:58.784008 5267 sys_catalog.cc:455] T 00000000000000000000000000000000 P af57b2dc49d84cc7a0fe7fd18be133a8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "af57b2dc49d84cc7a0fe7fd18be133a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af57b2dc49d84cc7a0fe7fd18be133a8" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 45351 } } }
I20260810 04:06:58.784273 5267 sys_catalog.cc:458] T 00000000000000000000000000000000 P af57b2dc49d84cc7a0fe7fd18be133a8 [sys.catalog]: This master's current role is: LEADER
I20260810 04:06:58.784363 5275 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 04:06:58.784847 5275 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 04:06:58.786299 5275 catalog_manager.cc:1347] Generated new cluster ID: 51bc7e80ee31445d9654e2f231f8c61e
I20260810 04:06:58.786350 5275 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 04:06:58.792527 5275 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 04:06:58.793007 5275 catalog_manager.cc:1504] Loading token signing keys...
I20260810 04:06:58.798180 5275 catalog_manager.cc:5868] T 00000000000000000000000000000000 P af57b2dc49d84cc7a0fe7fd18be133a8: Generated new TSK 0
I20260810 04:06:58.798384 5275 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 04:06:58.808416 5264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:58.804144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54752 (local address 127.31.32.62:45351)
0810 04:06:58.804204 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0810 04:06:58.804206 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:06:58.804327 (+ 121us) server_negotiation.cc:407] Connection header received
0810 04:06:58.804406 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:06:58.804406 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:06:58.804417 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:06:58.804472 (+ 55us) server_negotiation.cc:226] Negotiated authn=JWT
0810 04:06:58.804621 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:58.805089 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:58.806207 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:58.806414 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:58.806464 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:06:58.806669 (+ 205us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0810 04:06:58.808105 (+ 1436us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38203/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0810 04:06:58.808319 (+ 214us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:54752: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38203/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":31}
W20260810 04:06:58.808494 5293 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:06:58.804064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:45351 (local address 127.0.0.1:54752)
0810 04:06:58.804248 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0810 04:06:58.804260 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 04:06:58.804327 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:06:58.804481 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:06:58.804482 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:06:58.804484 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0810 04:06:58.804553 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:06:58.804555 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:06:58.805187 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:06:58.805188 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:06:58.806123 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:06:58.806126 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:06:58.806495 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:06:58.806496 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:06:58.806556 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:06:58.806562 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0810 04:06:58.808265 (+ 1703us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38203/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0810 04:06:58.808382 (+ 117us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.31.32.62:45351: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38203/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":133,"threads_started":1}
I20260810 04:06:58.808768 31872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskws89Io/build/release/bin/kudu with pid 5190
E20260810 04:06:58.821259 5294 webserver.cc:533] Webserver: sslize failed:
E20260810 04:06:58.821322 5294 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-08-10T04:06:58Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (203 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-08-10T04:06:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T04:06:58Z Disabled control of system clock
I20260810 04:06:58.862681 31872 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskws89Io/build/release/bin/kudu
/tmp/dist-test-taskws89Io/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786334794676589-31872-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786334794676589-31872-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786334794676589-31872-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786334794676589-31872-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.32.62:36309
--webserver_interface=127.31.32.62
--webserver_port=0
--builtin_ntp_servers=127.31.32.20:46021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.32.62:36309
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786334794676589-31872-0/testchainca.cert with env {}
W20260810 04:06:58.943742 5298 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:06:58.943964 5298 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:06:58.944001 5298 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:06:58.945219 5298 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:06:58.945290 5298 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:06:58.945315 5298 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:06:58.945345 5298 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:06:58.945372 5298 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260810 04:06:58.945398 5298 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260810 04:06:58.945425 5298 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260810 04:06:58.946936 5298 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.32.20:46021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786334794676589-31872-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786334794676589-31872-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.32.62:36309
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.32.62:36309
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786334794676589-31872-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.32.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786334794676589-31872-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5298
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786334794676589-31872-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:06:58.947216 5298 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:06:58.947422 5298 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:06:58.952298 5304 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:06:58.952420 5305 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:06:58.952520 5298 server_base.cc:1034] running on GCE node
W20260810 04:06:58.952358 5307 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:06:58.952862 5298 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:06:58.953118 5298 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:06:58.954258 5298 hybrid_clock.cc:648] HybridClock initialized: now 1786334818954247 us; error 30 us; skew 500 ppm
I20260810 04:06:58.955341 5298 webserver.cc:460] Webserver started at http://127.31.32.62:34757/ using document root <none> and password file <none>
I20260810 04:06:58.955542 5298 fs_manager.cc:359] Metadata directory not provided
I20260810 04:06:58.955585 5298 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:06:58.955667 5298 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:06:58.956792 5298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786334794676589-31872-0/minicluster-data/master-0/data/instance:
uuid: "ad574f05ee664645bc97cfc6d7c483f0"
format_stamp: "Formatted at 2026-08-10 04:06:58 on dist-test-slave-8w9v"
server_key: "5871f2c13eaaf27d99afd4792854598c"
server_key_iv: "8b723faf0c9eb1dfcfcbea0d957e5064"
server_key_version: "encryptionkey@0"
I20260810 04:06:58.957188 5298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786334794676589-31872-0/minicluster-data/master-0/wal/instance:
uuid: "ad574f05ee664645bc97cfc6d7c483f0"
format_stamp: "Formatted at 2026-08-10 04:06:58 on dist-test-slave-8w9v"
server_key: "5871f2c13eaaf27d99afd4792854598c"
server_key_iv: "8b723faf0c9eb1dfcfcbea0d957e5064"
server_key_version: "encryptionkey@0"
I20260810 04:06:58.958606 5298 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 04:06:58.959277 5313 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:06:58.959463 5298 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 04:06:58.959556 5298 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786334794676589-31872-0/minicluster-data/master-0/data,/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786334794676589-31872-0/minicluster-data/master-0/wal
uuid: "ad574f05ee664645bc97cfc6d7c483f0"
format_stamp: "Formatted at 2026-08-10 04:06:58 on dist-test-slave-8w9v"
server_key: "5871f2c13eaaf27d99afd4792854598c"
server_key_iv: "8b723faf0c9eb1dfcfcbea0d957e5064"
server_key_version: "encryptionkey@0"
I20260810 04:06:58.959626 5298 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786334794676589-31872-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786334794676589-31872-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786334794676589-31872-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:06:58.972568 5298 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:06:58.972867 5298 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:06:58.973033 5298 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:06:58.977519 5298 rpc_server.cc:307] RPC server started. Bound to: 127.31.32.62:36309
I20260810 04:06:58.977592 5368 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.32.62:36309 every 8 connection(s)
I20260810 04:06:58.977905 5298 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786334794676589-31872-0/minicluster-data/master-0/data/info.pb
I20260810 04:06:58.978853 5370 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:06:58.981086 5370 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad574f05ee664645bc97cfc6d7c483f0: Bootstrap starting.
I20260810 04:06:58.981657 5370 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ad574f05ee664645bc97cfc6d7c483f0: Neither blocks nor log segments found. Creating new log.
I20260810 04:06:58.981909 5370 log.cc:824] T 00000000000000000000000000000000 P ad574f05ee664645bc97cfc6d7c483f0: Log is configured to *not* fsync() on all Append() calls
I20260810 04:06:58.982537 5370 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad574f05ee664645bc97cfc6d7c483f0: No bootstrap required, opened a new log
I20260810 04:06:58.984043 5370 raft_consensus.cc:348] T 00000000000000000000000000000000 P ad574f05ee664645bc97cfc6d7c483f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad574f05ee664645bc97cfc6d7c483f0" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 36309 } }
I20260810 04:06:58.984261 5370 raft_consensus.cc:374] T 00000000000000000000000000000000 P ad574f05ee664645bc97cfc6d7c483f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:06:58.984299 5370 raft_consensus.cc:729] T 00000000000000000000000000000000 P ad574f05ee664645bc97cfc6d7c483f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad574f05ee664645bc97cfc6d7c483f0, State: Initialized, Role: FOLLOWER
I20260810 04:06:58.984400 5370 consensus_queue.cc:260] T 00000000000000000000000000000000 P ad574f05ee664645bc97cfc6d7c483f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad574f05ee664645bc97cfc6d7c483f0" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 36309 } }
I20260810 04:06:58.984476 5370 raft_consensus.cc:388] T 00000000000000000000000000000000 P ad574f05ee664645bc97cfc6d7c483f0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 04:06:58.984513 5370 raft_consensus.cc:482] T 00000000000000000000000000000000 P ad574f05ee664645bc97cfc6d7c483f0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 04:06:58.984562 5370 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ad574f05ee664645bc97cfc6d7c483f0 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:06:58.985109 5370 raft_consensus.cc:504] T 00000000000000000000000000000000 P ad574f05ee664645bc97cfc6d7c483f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad574f05ee664645bc97cfc6d7c483f0" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 36309 } }
I20260810 04:06:58.985240 5370 leader_election.cc:302] T 00000000000000000000000000000000 P ad574f05ee664645bc97cfc6d7c483f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ad574f05ee664645bc97cfc6d7c483f0; no voters:
I20260810 04:06:58.985425 5370 leader_election.cc:288] T 00000000000000000000000000000000 P ad574f05ee664645bc97cfc6d7c483f0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 04:06:58.985471 5373 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ad574f05ee664645bc97cfc6d7c483f0 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:06:58.985630 5373 raft_consensus.cc:686] T 00000000000000000000000000000000 P ad574f05ee664645bc97cfc6d7c483f0 [term 1 LEADER]: Becoming Leader. State: Replica: ad574f05ee664645bc97cfc6d7c483f0, State: Running, Role: LEADER
I20260810 04:06:58.985685 5370 sys_catalog.cc:564] T 00000000000000000000000000000000 P ad574f05ee664645bc97cfc6d7c483f0 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:06:58.985728 5373 consensus_queue.cc:237] T 00000000000000000000000000000000 P ad574f05ee664645bc97cfc6d7c483f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad574f05ee664645bc97cfc6d7c483f0" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 36309 } }
I20260810 04:06:58.986115 5374 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad574f05ee664645bc97cfc6d7c483f0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ad574f05ee664645bc97cfc6d7c483f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad574f05ee664645bc97cfc6d7c483f0" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 36309 } } }
I20260810 04:06:58.986230 5375 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad574f05ee664645bc97cfc6d7c483f0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ad574f05ee664645bc97cfc6d7c483f0. Latest consensus state: current_term: 1 leader_uuid: "ad574f05ee664645bc97cfc6d7c483f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad574f05ee664645bc97cfc6d7c483f0" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 36309 } } }
I20260810 04:06:58.986320 5375 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad574f05ee664645bc97cfc6d7c483f0 [sys.catalog]: This master's current role is: LEADER
I20260810 04:06:58.986219 5374 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad574f05ee664645bc97cfc6d7c483f0 [sys.catalog]: This master's current role is: LEADER
I20260810 04:06:58.986526 5382 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 04:06:58.987035 31872 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskws89Io/build/release/bin/kudu as pid 5298
I20260810 04:06:58.987134 31872 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786334794676589-31872-0/minicluster-data/master-0/wal/instance
I20260810 04:06:58.987322 31872 external_mini_cluster.cc:1428] Setting key 725bd8eb1480d857b385fe53027e73a6
I20260810 04:06:58.987916 5382 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 04:06:58.989461 5382 catalog_manager.cc:1347] Generated new cluster ID: 57e0bdd31ec24e0d9b643766316a98b4
I20260810 04:06:58.989516 5382 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 04:06:58.993620 5391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:58.987735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52432 (local address 127.31.32.62:36309)
0810 04:06:58.988145 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0810 04:06:58.988155 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 04:06:58.988188 (+ 33us) server_negotiation.cc:407] Connection header received
0810 04:06:58.988539 (+ 351us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:06:58.988548 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:06:58.988626 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:06:58.988772 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:06:58.988995 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:58.990360 (+ 1365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:58.991487 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:58.992101 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:58.992231 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:06:58.992941 (+ 710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:06:58.992945 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:06:58.992945 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:06:58.993138 (+ 193us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:06:58.993189 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:06:58.993202 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:06:58.993264 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0810 04:06:58.993362 (+ 98us) server_negotiation.cc:299] Negotiation successful
0810 04:06:58.993430 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":82,"threads_started":1}
I20260810 04:06:58.994534 31872 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 04:06:59.005082 5382 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 04:06:59.005621 5382 catalog_manager.cc:1504] Loading token signing keys...
I20260810 04:06:59.009223 5382 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ad574f05ee664645bc97cfc6d7c483f0: Generated new TSK 0
I20260810 04:06:59.009403 5382 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 04:06:59.041419 5391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:06:59.018839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52444 (local address 127.31.32.62:36309)
0810 04:06:59.018907 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0810 04:06:59.018909 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:06:59.019026 (+ 117us) server_negotiation.cc:407] Connection header received
0810 04:06:59.019103 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:06:59.019104 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:06:59.019127 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:06:59.019217 (+ 90us) server_negotiation.cc:226] Negotiated authn=JWT
0810 04:06:59.019355 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:59.019863 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:59.020752 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:06:59.020983 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:06:59.021071 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:06:59.021257 (+ 186us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0810 04:06:59.041146 (+ 19889us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0810 04:06:59.041345 (+ 199us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:52444: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":41}
W20260810 04:06:59.041502 5391 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:52444: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260810 04:06:59.041519 5399 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:06:59.018761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:36309 (local address 127.0.0.1:52444)
0810 04:06:59.018950 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0810 04:06:59.018960 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 04:06:59.019041 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:06:59.019216 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:06:59.019217 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:06:59.019219 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0810 04:06:59.019279 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:06:59.019281 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:06:59.019964 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:06:59.019965 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:06:59.020666 (+ 701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:06:59.020668 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:06:59.021089 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:06:59.021090 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:06:59.021147 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:06:59.021152 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0810 04:06:59.041285 (+ 20133us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0810 04:06:59.041404 (+ 119us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.31.32.62:36309: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":115,"threads_started":1}
I20260810 04:06:59.041810 31872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskws89Io/build/release/bin/kudu with pid 5298
2026-08-10T04:06:59Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (207 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (88 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (86 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786334794676589-31872-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 04:06:59 dist-test-slave-8w9v krb5kdc[5411](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 04:06:59 dist-test-slave-8w9v krb5kdc[5411](info): set up 2 sockets
Aug 10 04:06:59 dist-test-slave-8w9v krb5kdc[5411](info): commencing operation
krb5kdc: starting...
W20260810 04:07:00.311959 31872 mini_kdc.cc:121] Time spent starting KDC: real 1.075s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786334794676589-31872-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786334794676589-31872-0/krb5kdc/test-user.keytab.
Aug 10 04:07:00 dist-test-slave-8w9v krb5kdc[5411](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334820, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T04:07:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T04:07:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.32.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.32.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.32.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.62.keytab.
Entry for principal kudu/127.31.32.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.62.keytab.
I20260810 04:07:00.431236 31872 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskws89Io/build/release/bin/kudu
/tmp/dist-test-taskws89Io/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786334794676589-31872-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786334794676589-31872-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786334794676589-31872-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786334794676589-31872-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.32.62:36977
--webserver_interface=127.31.32.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.62.keytab
--principal=kudu/127.31.32.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.32.20:35521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.32.62:36977
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786334794676589-31872-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786334794676589-31872-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786334794676589-31872-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:07:00.512439 5424 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:07:00.512655 5424 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:07:00.512691 5424 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:07:00.513820 5424 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:07:00.513888 5424 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:07:00.513912 5424 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:07:00.513934 5424 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:07:00.513962 5424 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:07:00.515377 5424 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.32.20:35521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786334794676589-31872-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786334794676589-31872-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.32.62:36977
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.62.keytab
--principal=kudu/127.31.32.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.32.62:36977
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786334794676589-31872-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.32.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5424
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786334794676589-31872-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:07:00.515642 5424 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:07:00.515865 5424 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:07:00.520586 5429 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:07:00.520696 5424 server_base.cc:1034] running on GCE node
W20260810 04:07:00.520613 5432 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:07:00.520653 5430 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:07:00.521034 5424 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:07:00.521332 5424 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:07:00.522502 5424 hybrid_clock.cc:648] HybridClock initialized: now 1786334820522477 us; error 40 us; skew 500 ppm
Aug 10 04:07:00 dist-test-slave-8w9v krb5kdc[5411](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334820, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:07:00.528961 5424 init.cc:377] Logged in from keytab as kudu/127.31.32.62@KRBTEST.COM (short username kudu)
I20260810 04:07:00.529389 5424 webserver.cc:460] Webserver started at http://127.31.32.62:34687/ using document root <none> and password file <none>
I20260810 04:07:00.529633 5424 fs_manager.cc:359] Metadata directory not provided
I20260810 04:07:00.529702 5424 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:07:00.529856 5424 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:07:00.530872 5424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786334794676589-31872-0/minicluster-data/master-0/data/instance:
uuid: "63819d9101ad4e188faba387fe94617e"
format_stamp: "Formatted at 2026-08-10 04:07:00 on dist-test-slave-8w9v"
server_key: "1c62df84cbbc7bf018b89c6da3e6fb37"
server_key_iv: "5e874d79f2a233725225a1a7f8b33816"
server_key_version: "encryptionkey@0"
I20260810 04:07:00.531234 5424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786334794676589-31872-0/minicluster-data/master-0/wal/instance:
uuid: "63819d9101ad4e188faba387fe94617e"
format_stamp: "Formatted at 2026-08-10 04:07:00 on dist-test-slave-8w9v"
server_key: "1c62df84cbbc7bf018b89c6da3e6fb37"
server_key_iv: "5e874d79f2a233725225a1a7f8b33816"
server_key_version: "encryptionkey@0"
I20260810 04:07:00.532656 5424 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260810 04:07:00.533401 5440 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:07:00.533571 5424 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 04:07:00.533674 5424 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786334794676589-31872-0/minicluster-data/master-0/data,/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786334794676589-31872-0/minicluster-data/master-0/wal
uuid: "63819d9101ad4e188faba387fe94617e"
format_stamp: "Formatted at 2026-08-10 04:07:00 on dist-test-slave-8w9v"
server_key: "1c62df84cbbc7bf018b89c6da3e6fb37"
server_key_iv: "5e874d79f2a233725225a1a7f8b33816"
server_key_version: "encryptionkey@0"
I20260810 04:07:00.533754 5424 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786334794676589-31872-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786334794676589-31872-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786334794676589-31872-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:07:00.546918 5424 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:07:00.548774 5424 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:07:00.548946 5424 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:07:00.552906 5424 rpc_server.cc:307] RPC server started. Bound to: 127.31.32.62:36977
I20260810 04:07:00.552937 5496 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.32.62:36977 every 8 connection(s)
I20260810 04:07:00.553270 5424 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786334794676589-31872-0/minicluster-data/master-0/data/info.pb
I20260810 04:07:00.554127 5497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:07:00.555806 31872 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskws89Io/build/release/bin/kudu as pid 5424
I20260810 04:07:00.555909 31872 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786334794676589-31872-0/minicluster-data/master-0/wal/instance
I20260810 04:07:00.556111 31872 external_mini_cluster.cc:1428] Setting key 3648f5aee19651da3292b64789ccd11d
I20260810 04:07:00.556845 5497 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63819d9101ad4e188faba387fe94617e: Bootstrap starting.
I20260810 04:07:00.557462 5497 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 63819d9101ad4e188faba387fe94617e: Neither blocks nor log segments found. Creating new log.
I20260810 04:07:00.557852 5497 log.cc:824] T 00000000000000000000000000000000 P 63819d9101ad4e188faba387fe94617e: Log is configured to *not* fsync() on all Append() calls
I20260810 04:07:00.558568 5497 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63819d9101ad4e188faba387fe94617e: No bootstrap required, opened a new log
I20260810 04:07:00.560740 5497 raft_consensus.cc:348] T 00000000000000000000000000000000 P 63819d9101ad4e188faba387fe94617e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63819d9101ad4e188faba387fe94617e" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 36977 } }
I20260810 04:07:00.560915 5497 raft_consensus.cc:374] T 00000000000000000000000000000000 P 63819d9101ad4e188faba387fe94617e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:07:00.560956 5497 raft_consensus.cc:729] T 00000000000000000000000000000000 P 63819d9101ad4e188faba387fe94617e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63819d9101ad4e188faba387fe94617e, State: Initialized, Role: FOLLOWER
I20260810 04:07:00.561088 5497 consensus_queue.cc:260] T 00000000000000000000000000000000 P 63819d9101ad4e188faba387fe94617e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63819d9101ad4e188faba387fe94617e" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 36977 } }
I20260810 04:07:00.561201 5497 raft_consensus.cc:388] T 00000000000000000000000000000000 P 63819d9101ad4e188faba387fe94617e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 04:07:00.561247 5497 raft_consensus.cc:482] T 00000000000000000000000000000000 P 63819d9101ad4e188faba387fe94617e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 04:07:00.561305 5497 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 63819d9101ad4e188faba387fe94617e [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:07:00.562063 5497 raft_consensus.cc:504] T 00000000000000000000000000000000 P 63819d9101ad4e188faba387fe94617e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63819d9101ad4e188faba387fe94617e" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 36977 } }
I20260810 04:07:00.562193 5497 leader_election.cc:302] T 00000000000000000000000000000000 P 63819d9101ad4e188faba387fe94617e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 63819d9101ad4e188faba387fe94617e; no voters:
I20260810 04:07:00.562350 5497 leader_election.cc:288] T 00000000000000000000000000000000 P 63819d9101ad4e188faba387fe94617e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 04:07:00.562386 5503 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 63819d9101ad4e188faba387fe94617e [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:07:00.562552 5503 raft_consensus.cc:686] T 00000000000000000000000000000000 P 63819d9101ad4e188faba387fe94617e [term 1 LEADER]: Becoming Leader. State: Replica: 63819d9101ad4e188faba387fe94617e, State: Running, Role: LEADER
I20260810 04:07:00.562615 5497 sys_catalog.cc:564] T 00000000000000000000000000000000 P 63819d9101ad4e188faba387fe94617e [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:07:00.562644 5503 consensus_queue.cc:237] T 00000000000000000000000000000000 P 63819d9101ad4e188faba387fe94617e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63819d9101ad4e188faba387fe94617e" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 36977 } }
I20260810 04:07:00.562985 5503 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63819d9101ad4e188faba387fe94617e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "63819d9101ad4e188faba387fe94617e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63819d9101ad4e188faba387fe94617e" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 36977 } } }
I20260810 04:07:00.563117 5503 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63819d9101ad4e188faba387fe94617e [sys.catalog]: This master's current role is: LEADER
Aug 10 04:07:00 dist-test-slave-8w9v krb5kdc[5411](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334820, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.32.62@KRBTEST.COM
I20260810 04:07:00.563352 5503 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63819d9101ad4e188faba387fe94617e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 63819d9101ad4e188faba387fe94617e. Latest consensus state: current_term: 1 leader_uuid: "63819d9101ad4e188faba387fe94617e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63819d9101ad4e188faba387fe94617e" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 36977 } } }
I20260810 04:07:00.563479 5503 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63819d9101ad4e188faba387fe94617e [sys.catalog]: This master's current role is: LEADER
I20260810 04:07:00.563907 5512 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 04:07:00.565099 5512 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 04:07:00.566839 5512 catalog_manager.cc:1347] Generated new cluster ID: fa011fb3b70642c8bcba307399488c19
I20260810 04:07:00.566898 5512 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 04:07:00.567643 5501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:00.556669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59132 (local address 127.31.32.62:36977)
0810 04:07:00.556845 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0810 04:07:00.556854 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 04:07:00.556890 (+ 36us) server_negotiation.cc:407] Connection header received
0810 04:07:00.557375 (+ 485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:07:00.557381 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:07:00.557425 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:07:00.557595 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:07:00.558361 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:00.559278 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:00.560411 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:00.560795 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:00.560873 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:00.563881 (+ 3008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:07:00.563885 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:07:00.563886 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:07:00.563898 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:07:00.565969 (+ 2071us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:07:00.566388 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:07:00.566392 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:07:00.566392 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:07:00.566499 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:07:00.566744 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:07:00.566745 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:07:00.566745 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:07:00.567065 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:07:00.567223 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0810 04:07:00.567340 (+ 117us) server_negotiation.cc:299] Negotiation successful
0810 04:07:00.567476 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":96,"threads_started":1}
I20260810 04:07:00.596320 5512 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 04:07:00.597083 5512 catalog_manager.cc:1504] Loading token signing keys...
I20260810 04:07:00.604581 5512 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 63819d9101ad4e188faba387fe94617e: Generated new TSK 0
I20260810 04:07:00.604769 5512 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.32.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.32.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.32.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.1.keytab.
Entry for principal kudu/127.31.32.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.1.keytab.
I20260810 04:07:00.641319 31872 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskws89Io/build/release/bin/kudu
/tmp/dist-test-taskws89Io/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786334794676589-31872-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786334794676589-31872-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786334794676589-31872-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786334794676589-31872-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.32.1:0
--local_ip_for_outbound_sockets=127.31.32.1
--webserver_interface=127.31.32.1
--webserver_port=0
--tserver_master_addrs=127.31.32.62:36977
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.1.keytab
--principal=kudu/127.31.32.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.32.20:35521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786334794676589-31872-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786334794676589-31872-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786334794676589-31872-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:07:00.723189 5527 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:07:00.723392 5527 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:07:00.723414 5527 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:07:00.724607 5527 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:07:00.724648 5527 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:07:00.724674 5527 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.32.1
I20260810 04:07:00.726291 5527 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.32.20:35521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786334794676589-31872-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786334794676589-31872-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.1.keytab
--principal=kudu/127.31.32.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.32.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786334794676589-31872-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.32.1
--webserver_port=0
--tserver_master_addrs=127.31.32.62:36977
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5527
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.32.1
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786334794676589-31872-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:07:00.726528 5527 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:07:00.726775 5527 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:07:00.731693 5534 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:07:00.731689 5533 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:07:00.731689 5536 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:07:00.732088 5527 server_base.cc:1034] running on GCE node
I20260810 04:07:00.732359 5527 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:07:00.732604 5527 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:07:00.733768 5527 hybrid_clock.cc:648] HybridClock initialized: now 1786334820733750 us; error 30 us; skew 500 ppm
Aug 10 04:07:00 dist-test-slave-8w9v krb5kdc[5411](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334820, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:07:00.736315 5527 init.cc:377] Logged in from keytab as kudu/127.31.32.1@KRBTEST.COM (short username kudu)
I20260810 04:07:00.736763 5527 webserver.cc:460] Webserver started at http://127.31.32.1:42763/ using document root <none> and password file <none>
I20260810 04:07:00.737035 5527 fs_manager.cc:359] Metadata directory not provided
I20260810 04:07:00.737107 5527 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:07:00.737247 5527 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:07:00.738430 5527 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786334794676589-31872-0/minicluster-data/ts-0/data/instance:
uuid: "8bcaccdfca05437499c301abe69fef92"
format_stamp: "Formatted at 2026-08-10 04:07:00 on dist-test-slave-8w9v"
server_key: "56ce05bb71df4f8ff0378eab87cbe557"
server_key_iv: "e584f51741ac4702fb279ea95aa227ed"
server_key_version: "encryptionkey@0"
I20260810 04:07:00.738862 5527 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786334794676589-31872-0/minicluster-data/ts-0/wal/instance:
uuid: "8bcaccdfca05437499c301abe69fef92"
format_stamp: "Formatted at 2026-08-10 04:07:00 on dist-test-slave-8w9v"
server_key: "56ce05bb71df4f8ff0378eab87cbe557"
server_key_iv: "e584f51741ac4702fb279ea95aa227ed"
server_key_version: "encryptionkey@0"
I20260810 04:07:00.740437 5527 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.001s
I20260810 04:07:00.741154 5544 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:07:00.741320 5527 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260810 04:07:00.741441 5527 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786334794676589-31872-0/minicluster-data/ts-0/data,/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786334794676589-31872-0/minicluster-data/ts-0/wal
uuid: "8bcaccdfca05437499c301abe69fef92"
format_stamp: "Formatted at 2026-08-10 04:07:00 on dist-test-slave-8w9v"
server_key: "56ce05bb71df4f8ff0378eab87cbe557"
server_key_iv: "e584f51741ac4702fb279ea95aa227ed"
server_key_version: "encryptionkey@0"
I20260810 04:07:00.741513 5527 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786334794676589-31872-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786334794676589-31872-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786334794676589-31872-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:07:00.759322 5527 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:07:00.761206 5527 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:07:00.761405 5527 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:07:00.761649 5527 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:07:00.761981 5527 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:07:00.762039 5527 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:07:00.762079 5527 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:07:00.762112 5527 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:07:00.768905 5527 rpc_server.cc:307] RPC server started. Bound to: 127.31.32.1:40109
I20260810 04:07:00.768942 5667 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.32.1:40109 every 8 connection(s)
I20260810 04:07:00.769275 5527 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786334794676589-31872-0/minicluster-data/ts-0/data/info.pb
Aug 10 04:07:00 dist-test-slave-8w9v krb5kdc[5411](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334820, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.1@KRBTEST.COM for kudu/127.31.32.62@KRBTEST.COM
I20260810 04:07:00.776515 31872 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskws89Io/build/release/bin/kudu as pid 5527
I20260810 04:07:00.776633 31872 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786334794676589-31872-0/minicluster-data/ts-0/wal/instance
I20260810 04:07:00.776870 31872 external_mini_cluster.cc:1428] Setting key 7ce42f915bf565a5da1da481ade1cf7d
I20260810 04:07:00.778714 5501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:00.770077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.1:57461 (local address 127.31.32.62:36977)
0810 04:07:00.770140 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0810 04:07:00.770142 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:07:00.770351 (+ 209us) server_negotiation.cc:407] Connection header received
0810 04:07:00.771245 (+ 894us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:07:00.771246 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:07:00.771257 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:07:00.771314 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:07:00.771903 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:00.772485 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:00.773444 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:00.773639 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:00.773693 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:00.775696 (+ 2003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:07:00.775698 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:07:00.775698 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:07:00.775704 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:07:00.777280 (+ 1576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:07:00.777737 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:07:00.777738 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:07:00.777738 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:07:00.777805 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:07:00.778054 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:07:00.778055 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:07:00.778056 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:07:00.778175 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:07:00.778249 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0810 04:07:00.778537 (+ 288us) server_negotiation.cc:299] Negotiation successful
0810 04:07:00.778624 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260810 04:07:00.778898 5670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:00.770060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:36977 (local address 127.31.32.1:57461)
0810 04:07:00.770231 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0810 04:07:00.770265 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0810 04:07:00.771130 (+ 865us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:07:00.771393 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:07:00.771396 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:07:00.771628 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:07:00.771800 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:00.771804 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:00.772559 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:00.772560 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:00.773365 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:00.773367 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:00.773719 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:00.773720 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:00.773813 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:00.774244 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:07:00.774252 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:07:00.775608 (+ 1356us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:07:00.777363 (+ 1755us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:07:00.777369 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:07:00.777371 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:07:00.777651 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:07:00.777871 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:07:00.777872 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:07:00.777873 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:07:00.777984 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:07:00.778254 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:07:00.778262 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:07:00.778398 (+ 136us) client_negotiation.cc:769] Sending connection context
0810 04:07:00.778502 (+ 104us) client_negotiation.cc:240] Negotiation successful
0810 04:07:00.778653 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":93,"threads_started":1}
I20260810 04:07:00.779560 5668 heartbeater.cc:347] Connected to a master server at 127.31.32.62:36977
I20260810 04:07:00.779665 5668 heartbeater.cc:464] Registering TS with master...
I20260810 04:07:00.779839 5668 heartbeater.cc:511] Master 127.31.32.62:36977 requested a full tablet report, sending...
I20260810 04:07:00.780437 5459 ts_manager.cc:194] Registered new tserver with Master: 8bcaccdfca05437499c301abe69fef92 (127.31.32.1:40109)
I20260810 04:07:00.781399 5459 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.32.1@KRBTEST.COM'} at 127.31.32.1:57461
WARNING: no policy specified for kudu/127.31.32.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.32.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.32.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.2.keytab.
Entry for principal kudu/127.31.32.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.2.keytab.
I20260810 04:07:00.802847 31872 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskws89Io/build/release/bin/kudu
/tmp/dist-test-taskws89Io/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786334794676589-31872-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786334794676589-31872-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786334794676589-31872-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786334794676589-31872-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.32.2:0
--local_ip_for_outbound_sockets=127.31.32.2
--webserver_interface=127.31.32.2
--webserver_port=0
--tserver_master_addrs=127.31.32.62:36977
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.2.keytab
--principal=kudu/127.31.32.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.32.20:35521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786334794676589-31872-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786334794676589-31872-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786334794676589-31872-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:07:00.884387 5673 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:07:00.884569 5673 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:07:00.884591 5673 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:07:00.885627 5673 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:07:00.885658 5673 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:07:00.885682 5673 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.32.2
I20260810 04:07:00.887125 5673 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.32.20:35521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786334794676589-31872-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786334794676589-31872-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.2.keytab
--principal=kudu/127.31.32.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.32.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786334794676589-31872-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.32.2
--webserver_port=0
--tserver_master_addrs=127.31.32.62:36977
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5673
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.32.2
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786334794676589-31872-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:07:00.887310 5673 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:07:00.887481 5673 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 04:07:00.892427 5673 server_base.cc:1034] running on GCE node
W20260810 04:07:00.892369 5679 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:07:00.892400 5678 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:07:00.892354 5682 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:07:00.892812 5673 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:07:00.893031 5673 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:07:00.894186 5673 hybrid_clock.cc:648] HybridClock initialized: now 1786334820894183 us; error 41 us; skew 500 ppm
Aug 10 04:07:00 dist-test-slave-8w9v krb5kdc[5411](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334820, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:07:00.896798 5673 init.cc:377] Logged in from keytab as kudu/127.31.32.2@KRBTEST.COM (short username kudu)
I20260810 04:07:00.897255 5673 webserver.cc:460] Webserver started at http://127.31.32.2:45953/ using document root <none> and password file <none>
I20260810 04:07:00.897485 5673 fs_manager.cc:359] Metadata directory not provided
I20260810 04:07:00.897526 5673 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:07:00.897709 5673 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:07:00.898716 5673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786334794676589-31872-0/minicluster-data/ts-1/data/instance:
uuid: "ca2fe3fa528f4a1d9d328cb79ec0b020"
format_stamp: "Formatted at 2026-08-10 04:07:00 on dist-test-slave-8w9v"
server_key: "00059cccebb4faae417f629f09a1322a"
server_key_iv: "975d76cba0aa4febda4f576730c1e2df"
server_key_version: "encryptionkey@0"
I20260810 04:07:00.899070 5673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786334794676589-31872-0/minicluster-data/ts-1/wal/instance:
uuid: "ca2fe3fa528f4a1d9d328cb79ec0b020"
format_stamp: "Formatted at 2026-08-10 04:07:00 on dist-test-slave-8w9v"
server_key: "00059cccebb4faae417f629f09a1322a"
server_key_iv: "975d76cba0aa4febda4f576730c1e2df"
server_key_version: "encryptionkey@0"
I20260810 04:07:00.900382 5673 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260810 04:07:00.901081 5689 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:07:00.901252 5673 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 04:07:00.901364 5673 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786334794676589-31872-0/minicluster-data/ts-1/data,/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786334794676589-31872-0/minicluster-data/ts-1/wal
uuid: "ca2fe3fa528f4a1d9d328cb79ec0b020"
format_stamp: "Formatted at 2026-08-10 04:07:00 on dist-test-slave-8w9v"
server_key: "00059cccebb4faae417f629f09a1322a"
server_key_iv: "975d76cba0aa4febda4f576730c1e2df"
server_key_version: "encryptionkey@0"
I20260810 04:07:00.901437 5673 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786334794676589-31872-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786334794676589-31872-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786334794676589-31872-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:07:00.927603 5673 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:07:00.930017 5673 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:07:00.930178 5673 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:07:00.930426 5673 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:07:00.930737 5673 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:07:00.930805 5673 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:07:00.930858 5673 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:07:00.930886 5673 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:07:00.937119 5673 rpc_server.cc:307] RPC server started. Bound to: 127.31.32.2:39551
I20260810 04:07:00.937179 5812 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.32.2:39551 every 8 connection(s)
I20260810 04:07:00.937486 5673 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786334794676589-31872-0/minicluster-data/ts-1/data/info.pb
I20260810 04:07:00.937763 31872 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskws89Io/build/release/bin/kudu as pid 5673
I20260810 04:07:00.937877 31872 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786334794676589-31872-0/minicluster-data/ts-1/wal/instance
I20260810 04:07:00.938102 31872 external_mini_cluster.cc:1428] Setting key 2a2fb6e6c19ed0846b5548b5238b1800
Aug 10 04:07:00 dist-test-slave-8w9v krb5kdc[5411](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334820, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.2@KRBTEST.COM for kudu/127.31.32.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.31.32.3@KRBTEST.COM; defaulting to no policy
I20260810 04:07:00.948776 5501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:00.938312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.2:37335 (local address 127.31.32.62:36977)
0810 04:07:00.938368 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0810 04:07:00.938371 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:07:00.938567 (+ 196us) server_negotiation.cc:407] Connection header received
0810 04:07:00.939718 (+ 1151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:07:00.939719 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:07:00.939732 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:07:00.939811 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:07:00.940745 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:00.941525 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:00.942629 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:00.942875 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:00.942975 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:00.945678 (+ 2703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:07:00.945679 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:07:00.945680 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:07:00.945688 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:07:00.947284 (+ 1596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:07:00.947748 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:07:00.947750 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:07:00.947750 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:07:00.947813 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:07:00.948037 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:07:00.948038 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:07:00.948038 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:07:00.948184 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:07:00.948247 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0810 04:07:00.948580 (+ 333us) server_negotiation.cc:299] Negotiation successful
0810 04:07:00.948672 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260810 04:07:00.948966 5815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:00.938256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:36977 (local address 127.31.32.2:37335)
0810 04:07:00.938419 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0810 04:07:00.938471 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0810 04:07:00.939592 (+ 1121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:07:00.939924 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:07:00.939928 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:07:00.940340 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:07:00.940647 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:00.940655 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:00.941623 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:00.941624 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:00.942516 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:00.942518 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:00.942998 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:00.942999 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:00.943197 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:00.943786 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:07:00.943796 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:07:00.945567 (+ 1771us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:07:00.947369 (+ 1802us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:07:00.947375 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:07:00.947376 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:07:00.947672 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:07:00.947885 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:07:00.947886 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:07:00.947886 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:07:00.947986 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:07:00.948251 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:07:00.948255 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:07:00.948382 (+ 127us) client_negotiation.cc:769] Sending connection context
0810 04:07:00.948536 (+ 154us) client_negotiation.cc:240] Negotiation successful
0810 04:07:00.948708 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":90,"threads_started":1}
I20260810 04:07:00.949461 5813 heartbeater.cc:347] Connected to a master server at 127.31.32.62:36977
I20260810 04:07:00.949561 5813 heartbeater.cc:464] Registering TS with master...
I20260810 04:07:00.949767 5813 heartbeater.cc:511] Master 127.31.32.62:36977 requested a full tablet report, sending...
I20260810 04:07:00.950208 5459 ts_manager.cc:194] Registered new tserver with Master: ca2fe3fa528f4a1d9d328cb79ec0b020 (127.31.32.2:39551)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.32.3@KRBTEST.COM" created.
I20260810 04:07:00.950667 5459 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.32.2@KRBTEST.COM'} at 127.31.32.2:37335
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.32.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.3.keytab.
Entry for principal kudu/127.31.32.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.3.keytab.
I20260810 04:07:00.966586 31872 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskws89Io/build/release/bin/kudu
/tmp/dist-test-taskws89Io/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786334794676589-31872-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786334794676589-31872-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786334794676589-31872-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786334794676589-31872-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.32.3:0
--local_ip_for_outbound_sockets=127.31.32.3
--webserver_interface=127.31.32.3
--webserver_port=0
--tserver_master_addrs=127.31.32.62:36977
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.3.keytab
--principal=kudu/127.31.32.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.32.20:35521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786334794676589-31872-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786334794676589-31872-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786334794676589-31872-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:07:01.048992 5818 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:07:01.049189 5818 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:07:01.049304 5818 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:07:01.050487 5818 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:07:01.050529 5818 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:07:01.050591 5818 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.32.3
I20260810 04:07:01.052168 5818 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.32.20:35521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786334794676589-31872-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786334794676589-31872-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.3.keytab
--principal=kudu/127.31.32.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.32.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786334794676589-31872-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.32.3
--webserver_port=0
--tserver_master_addrs=127.31.32.62:36977
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5818
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.32.3
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786334794676589-31872-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:07:01.052436 5818 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:07:01.052663 5818 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:07:01.057704 5823 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:07:01.057709 5824 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:07:01.057719 5818 server_base.cc:1034] running on GCE node
W20260810 04:07:01.057704 5827 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:07:01.058178 5818 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:07:01.058375 5818 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:07:01.059540 5818 hybrid_clock.cc:648] HybridClock initialized: now 1786334821059529 us; error 29 us; skew 500 ppm
Aug 10 04:07:01 dist-test-slave-8w9v krb5kdc[5411](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334821, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:07:01.062213 5818 init.cc:377] Logged in from keytab as kudu/127.31.32.3@KRBTEST.COM (short username kudu)
I20260810 04:07:01.062703 5818 webserver.cc:460] Webserver started at http://127.31.32.3:38829/ using document root <none> and password file <none>
I20260810 04:07:01.062951 5818 fs_manager.cc:359] Metadata directory not provided
I20260810 04:07:01.062997 5818 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:07:01.063139 5818 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:07:01.064409 5818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786334794676589-31872-0/minicluster-data/ts-2/data/instance:
uuid: "a26dd48d8f7a4c098d959c8f561a96aa"
format_stamp: "Formatted at 2026-08-10 04:07:01 on dist-test-slave-8w9v"
server_key: "8f277a82666b3ec26fd4f7d109d319fe"
server_key_iv: "fddc54dc46f0c1e3956df113803cbafe"
server_key_version: "encryptionkey@0"
I20260810 04:07:01.064759 5818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786334794676589-31872-0/minicluster-data/ts-2/wal/instance:
uuid: "a26dd48d8f7a4c098d959c8f561a96aa"
format_stamp: "Formatted at 2026-08-10 04:07:01 on dist-test-slave-8w9v"
server_key: "8f277a82666b3ec26fd4f7d109d319fe"
server_key_iv: "fddc54dc46f0c1e3956df113803cbafe"
server_key_version: "encryptionkey@0"
I20260810 04:07:01.066238 5818 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 04:07:01.066959 5836 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:07:01.067171 5818 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 04:07:01.067247 5818 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786334794676589-31872-0/minicluster-data/ts-2/data,/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786334794676589-31872-0/minicluster-data/ts-2/wal
uuid: "a26dd48d8f7a4c098d959c8f561a96aa"
format_stamp: "Formatted at 2026-08-10 04:07:01 on dist-test-slave-8w9v"
server_key: "8f277a82666b3ec26fd4f7d109d319fe"
server_key_iv: "fddc54dc46f0c1e3956df113803cbafe"
server_key_version: "encryptionkey@0"
I20260810 04:07:01.067342 5818 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786334794676589-31872-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786334794676589-31872-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786334794676589-31872-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:07:01.085894 5818 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:07:01.088253 5818 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:07:01.088407 5818 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:07:01.088676 5818 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:07:01.089028 5818 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:07:01.089066 5818 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:07:01.089088 5818 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:07:01.089100 5818 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:07:01.094691 5818 rpc_server.cc:307] RPC server started. Bound to: 127.31.32.3:33783
I20260810 04:07:01.094764 5956 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.32.3:33783 every 8 connection(s)
I20260810 04:07:01.095091 5818 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786334794676589-31872-0/minicluster-data/ts-2/data/info.pb
Aug 10 04:07:01 dist-test-slave-8w9v krb5kdc[5411](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334821, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.3@KRBTEST.COM for kudu/127.31.32.62@KRBTEST.COM
I20260810 04:07:01.101802 31872 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskws89Io/build/release/bin/kudu as pid 5818
I20260810 04:07:01.101900 31872 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786334794676589-31872-0/minicluster-data/ts-2/wal/instance
I20260810 04:07:01.102078 31872 external_mini_cluster.cc:1428] Setting key a50d50a84c4114e845feddfb23f933d4
I20260810 04:07:01.104529 5501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:01.095851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.3:52833 (local address 127.31.32.62:36977)
0810 04:07:01.095916 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0810 04:07:01.095924 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 04:07:01.096142 (+ 218us) server_negotiation.cc:407] Connection header received
0810 04:07:01.097125 (+ 983us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:07:01.097126 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:07:01.097138 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:07:01.097189 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:07:01.097761 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:01.098248 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:01.099202 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:01.099379 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:01.099428 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:01.101475 (+ 2047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:07:01.101476 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:07:01.101476 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:07:01.101482 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:07:01.103258 (+ 1776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:07:01.103591 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:07:01.103592 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:07:01.103593 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:07:01.103640 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:07:01.103843 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:07:01.103843 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:07:01.103843 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:07:01.103936 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:07:01.103983 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0810 04:07:01.104307 (+ 324us) server_negotiation.cc:299] Negotiation successful
0810 04:07:01.104418 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260810 04:07:01.104734 5959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:01.095844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:36977 (local address 127.31.32.3:52833)
0810 04:07:01.096006 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0810 04:07:01.096042 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0810 04:07:01.097004 (+ 962us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:07:01.097245 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:07:01.097248 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:07:01.097505 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:07:01.097689 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:01.097693 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:01.098324 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:01.098337 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:01.099130 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:01.099132 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:01.099449 (+ 317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:01.099450 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:01.099544 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:01.100027 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:07:01.100035 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:07:01.101396 (+ 1361us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:07:01.103319 (+ 1923us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:07:01.103325 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:07:01.103327 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:07:01.103531 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:07:01.103694 (+ 163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:07:01.103695 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:07:01.103695 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:07:01.103785 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:07:01.103988 (+ 203us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:07:01.103992 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:07:01.104177 (+ 185us) client_negotiation.cc:769] Sending connection context
0810 04:07:01.104280 (+ 103us) client_negotiation.cc:240] Negotiation successful
0810 04:07:01.104461 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":84,"threads_started":1}
I20260810 04:07:01.105237 5957 heartbeater.cc:347] Connected to a master server at 127.31.32.62:36977
I20260810 04:07:01.105350 5957 heartbeater.cc:464] Registering TS with master...
I20260810 04:07:01.105522 5957 heartbeater.cc:511] Master 127.31.32.62:36977 requested a full tablet report, sending...
I20260810 04:07:01.105867 5459 ts_manager.cc:194] Registered new tserver with Master: a26dd48d8f7a4c098d959c8f561a96aa (127.31.32.3:33783)
I20260810 04:07:01.106238 5459 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.32.3@KRBTEST.COM'} at 127.31.32.3:52833
I20260810 04:07:01.113864 31872 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786334794676589-31872-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786334794676589-31872-0/krb5kdc/test-admin.keytab.
Aug 10 04:07:01 dist-test-slave-8w9v krb5kdc[5411](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334821, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:07:01.130316 31872 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260810 04:07:01.131846 5964 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:07:01.131115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54154 (local address 127.31.32.1:40109)
0810 04:07:01.131302 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0810 04:07:01.131307 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 04:07:01.131321 (+ 14us) server_negotiation.cc:407] Connection header received
0810 04:07:01.131347 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:07:01.131348 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:07:01.131361 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:07:01.131424 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:07:01.131766 (+ 342us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54154: BlockingRecv error: recv got EOF from 127.0.0.1:54154 (error 108)
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":96,"threads_started":1}
W20260810 04:07:01.132076 31872 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786334794676589-31872-0/krb5kdc/krb5cc' of type FILE
Aug 10 04:07:01 dist-test-slave-8w9v krb5kdc[5411](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334821, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:07:01.133076 31872 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 10 04:07:01 dist-test-slave-8w9v krb5kdc[5411](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334821, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:07:01.133910 31872 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 10 04:07:01 dist-test-slave-8w9v krb5kdc[5411](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334821, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.32.1@KRBTEST.COM
I20260810 04:07:01.141508 5964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:01.134397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54170 (local address 127.31.32.1:40109)
0810 04:07:01.134475 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0810 04:07:01.134477 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:07:01.134527 (+ 50us) server_negotiation.cc:407] Connection header received
0810 04:07:01.134574 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:07:01.134575 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:07:01.134584 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:07:01.134631 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:07:01.135001 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:01.135695 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:01.136599 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:01.136909 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:01.137013 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:01.139057 (+ 2044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:07:01.139064 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:07:01.139064 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:07:01.139071 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:07:01.140469 (+ 1398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:07:01.140760 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:07:01.140762 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:07:01.140763 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:07:01.140824 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:07:01.141018 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:07:01.141018 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:07:01.141019 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:07:01.141134 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:07:01.141222 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0810 04:07:01.141299 (+ 77us) server_negotiation.cc:299] Negotiation successful
0810 04:07:01.141359 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
W20260810 04:07:01.143283 5964 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:07:01.142650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54184 (local address 127.31.32.1:40109)
0810 04:07:01.142688 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0810 04:07:01.142690 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:07:01.142701 (+ 11us) server_negotiation.cc:407] Connection header received
0810 04:07:01.142771 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:07:01.142772 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:07:01.142782 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:07:01.142843 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:07:01.143239 (+ 396us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54184: BlockingRecv error: recv got EOF from 127.0.0.1:54184 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260810 04:07:01.143385 31872 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786334794676589-31872-0/krb5kdc/krb5cc' of type FILE
Aug 10 04:07:01 dist-test-slave-8w9v krb5kdc[5411](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334821, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:07:01.144374 31872 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 10 04:07:01 dist-test-slave-8w9v krb5kdc[5411](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334821, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:07:01.145255 31872 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 10 04:07:01 dist-test-slave-8w9v krb5kdc[5411](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334821, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.32.1@KRBTEST.COM
I20260810 04:07:01.151664 5964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:01.145787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54192 (local address 127.31.32.1:40109)
0810 04:07:01.145868 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0810 04:07:01.145870 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:07:01.145914 (+ 44us) server_negotiation.cc:407] Connection header received
0810 04:07:01.145950 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:07:01.145951 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:07:01.145958 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:07:01.146004 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:07:01.146318 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:01.146765 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:01.147421 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:01.147610 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:01.147666 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:01.149489 (+ 1823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:07:01.149491 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:07:01.149491 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:07:01.149497 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:07:01.150808 (+ 1311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:07:01.151080 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:07:01.151080 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:07:01.151081 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:07:01.151126 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:07:01.151267 (+ 141us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:07:01.151267 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:07:01.151268 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:07:01.151337 (+ 69us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:07:01.151398 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0810 04:07:01.151478 (+ 80us) server_negotiation.cc:299] Negotiation successful
0810 04:07:01.151530 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260810 04:07:01.153335 5964 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:07:01.152759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54198 (local address 127.31.32.1:40109)
0810 04:07:01.152831 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0810 04:07:01.152833 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:07:01.152892 (+ 59us) server_negotiation.cc:407] Connection header received
0810 04:07:01.152983 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:07:01.152983 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:07:01.152992 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:07:01.153036 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:07:01.153295 (+ 259us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54198: BlockingRecv error: recv got EOF from 127.0.0.1:54198 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260810 04:07:01.153438 31872 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786334794676589-31872-0/krb5kdc/krb5cc' of type FILE
Aug 10 04:07:01 dist-test-slave-8w9v krb5kdc[5411](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334821, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:07:01.154275 31872 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 10 04:07:01 dist-test-slave-8w9v krb5kdc[5411](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334821, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:07:01.155153 31872 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 10 04:07:01 dist-test-slave-8w9v krb5kdc[5411](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334821, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.32.1@KRBTEST.COM
I20260810 04:07:01.162180 5964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:01.155669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54206 (local address 127.31.32.1:40109)
0810 04:07:01.155725 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0810 04:07:01.155727 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:07:01.155751 (+ 24us) server_negotiation.cc:407] Connection header received
0810 04:07:01.155843 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:07:01.155844 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:07:01.155852 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:07:01.155920 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:07:01.156354 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:01.156779 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:01.157462 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:01.157646 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:01.157723 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:01.159615 (+ 1892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:07:01.159616 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:07:01.159616 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:07:01.159623 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:07:01.161064 (+ 1441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:07:01.161417 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:07:01.161418 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:07:01.161419 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:07:01.161504 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:07:01.161708 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:07:01.161709 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:07:01.161709 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:07:01.161802 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:07:01.161912 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0810 04:07:01.162024 (+ 112us) server_negotiation.cc:299] Negotiation successful
0810 04:07:01.162086 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260810 04:07:01.162967 31872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskws89Io/build/release/bin/kudu with pid 5527
I20260810 04:07:01.170019 31872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskws89Io/build/release/bin/kudu with pid 5673
I20260810 04:07:01.177040 31872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskws89Io/build/release/bin/kudu with pid 5818
I20260810 04:07:01.182928 31872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskws89Io/build/release/bin/kudu with pid 5424
2026-08-10T04:07:01Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (1968 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 04:07:01 dist-test-slave-8w9v krb5kdc[5977](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 04:07:01 dist-test-slave-8w9v krb5kdc[5977](info): set up 2 sockets
Aug 10 04:07:01 dist-test-slave-8w9v krb5kdc[5977](info): commencing operation
krb5kdc: starting...
W20260810 04:07:02.292742 31872 mini_kdc.cc:121] Time spent starting KDC: real 1.087s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/krb5kdc/test-user.keytab.
Aug 10 04:07:02 dist-test-slave-8w9v krb5kdc[5977](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334822, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T04:07:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T04:07:02Z Disabled control of system clock
WARNING: no policy specified for oryx/127.31.32.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.32.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.32.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/krb5kdc/oryx_127.31.32.62.keytab.
Entry for principal oryx/127.31.32.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/krb5kdc/oryx_127.31.32.62.keytab.
I20260810 04:07:02.412261 31872 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskws89Io/build/release/bin/kudu
/tmp/dist-test-taskws89Io/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.32.62:35131
--webserver_interface=127.31.32.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/krb5kdc/oryx_127.31.32.62.keytab
--principal=oryx/127.31.32.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.32.20:35351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.32.62:35131
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:07:02.493515 5991 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:07:02.493708 5991 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:07:02.493731 5991 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:07:02.494874 5991 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:07:02.494915 5991 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:07:02.494928 5991 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:07:02.494941 5991 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:07:02.494997 5991 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:07:02.496496 5991 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.32.20:35351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.32.62:35131
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/krb5kdc/oryx_127.31.32.62.keytab
--principal=oryx/127.31.32.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.32.62:35131
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.32.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5991
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:07:02.496722 5991 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:07:02.496973 5991 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:07:02.502015 5997 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:07:02.501994 5998 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:07:02.502115 5991 server_base.cc:1034] running on GCE node
W20260810 04:07:02.501995 6000 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:07:02.502494 5991 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:07:02.502789 5991 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:07:02.503940 5991 hybrid_clock.cc:648] HybridClock initialized: now 1786334822503952 us; error 54 us; skew 500 ppm
Aug 10 04:07:02 dist-test-slave-8w9v krb5kdc[5977](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334822, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.32.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:07:02.506682 5991 init.cc:377] Logged in from keytab as oryx/127.31.32.62@KRBTEST.COM (short username oryx)
I20260810 04:07:02.507236 5991 webserver.cc:460] Webserver started at http://127.31.32.62:35957/ using document root <none> and password file <none>
I20260810 04:07:02.507530 5991 fs_manager.cc:359] Metadata directory not provided
I20260810 04:07:02.507611 5991 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:07:02.507747 5991 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:07:02.508864 5991 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/minicluster-data/master-0/data/instance:
uuid: "b858c2bd3c2346bf9ac073d925ee5ef1"
format_stamp: "Formatted at 2026-08-10 04:07:02 on dist-test-slave-8w9v"
server_key: "9daaa01c9d3fa8066b0b6836d24389d8"
server_key_iv: "9231b58da9291eb2c5dfe07669a60657"
server_key_version: "encryptionkey@0"
I20260810 04:07:02.509233 5991 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/minicluster-data/master-0/wal/instance:
uuid: "b858c2bd3c2346bf9ac073d925ee5ef1"
format_stamp: "Formatted at 2026-08-10 04:07:02 on dist-test-slave-8w9v"
server_key: "9daaa01c9d3fa8066b0b6836d24389d8"
server_key_iv: "9231b58da9291eb2c5dfe07669a60657"
server_key_version: "encryptionkey@0"
I20260810 04:07:02.510573 5991 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 04:07:02.511298 6008 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:07:02.511492 5991 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 04:07:02.511598 5991 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/minicluster-data/master-0/data,/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/minicluster-data/master-0/wal
uuid: "b858c2bd3c2346bf9ac073d925ee5ef1"
format_stamp: "Formatted at 2026-08-10 04:07:02 on dist-test-slave-8w9v"
server_key: "9daaa01c9d3fa8066b0b6836d24389d8"
server_key_iv: "9231b58da9291eb2c5dfe07669a60657"
server_key_version: "encryptionkey@0"
I20260810 04:07:02.511670 5991 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:07:02.522782 5991 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:07:02.524652 5991 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:07:02.524809 5991 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:07:02.528951 5991 rpc_server.cc:307] RPC server started. Bound to: 127.31.32.62:35131
I20260810 04:07:02.528995 6068 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.32.62:35131 every 8 connection(s)
I20260810 04:07:02.529343 5991 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/minicluster-data/master-0/data/info.pb
I20260810 04:07:02.530359 6069 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:07:02.532769 6069 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b858c2bd3c2346bf9ac073d925ee5ef1: Bootstrap starting.
I20260810 04:07:02.533391 6069 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b858c2bd3c2346bf9ac073d925ee5ef1: Neither blocks nor log segments found. Creating new log.
I20260810 04:07:02.533638 6069 log.cc:824] T 00000000000000000000000000000000 P b858c2bd3c2346bf9ac073d925ee5ef1: Log is configured to *not* fsync() on all Append() calls
I20260810 04:07:02.534324 6069 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b858c2bd3c2346bf9ac073d925ee5ef1: No bootstrap required, opened a new log
I20260810 04:07:02.535881 6069 raft_consensus.cc:348] T 00000000000000000000000000000000 P b858c2bd3c2346bf9ac073d925ee5ef1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b858c2bd3c2346bf9ac073d925ee5ef1" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 35131 } }
I20260810 04:07:02.536034 6069 raft_consensus.cc:374] T 00000000000000000000000000000000 P b858c2bd3c2346bf9ac073d925ee5ef1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:07:02.536060 6069 raft_consensus.cc:729] T 00000000000000000000000000000000 P b858c2bd3c2346bf9ac073d925ee5ef1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b858c2bd3c2346bf9ac073d925ee5ef1, State: Initialized, Role: FOLLOWER
I20260810 04:07:02.536226 6069 consensus_queue.cc:260] T 00000000000000000000000000000000 P b858c2bd3c2346bf9ac073d925ee5ef1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b858c2bd3c2346bf9ac073d925ee5ef1" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 35131 } }
I20260810 04:07:02.536296 6069 raft_consensus.cc:388] T 00000000000000000000000000000000 P b858c2bd3c2346bf9ac073d925ee5ef1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 04:07:02.536320 6069 raft_consensus.cc:482] T 00000000000000000000000000000000 P b858c2bd3c2346bf9ac073d925ee5ef1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 04:07:02.536351 6069 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b858c2bd3c2346bf9ac073d925ee5ef1 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:07:02.537218 31872 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskws89Io/build/release/bin/kudu as pid 5991
I20260810 04:07:02.537246 6069 raft_consensus.cc:504] T 00000000000000000000000000000000 P b858c2bd3c2346bf9ac073d925ee5ef1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b858c2bd3c2346bf9ac073d925ee5ef1" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 35131 } }
I20260810 04:07:02.537355 31872 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/minicluster-data/master-0/wal/instance
I20260810 04:07:02.537359 6069 leader_election.cc:302] T 00000000000000000000000000000000 P b858c2bd3c2346bf9ac073d925ee5ef1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b858c2bd3c2346bf9ac073d925ee5ef1; no voters:
I20260810 04:07:02.537544 6069 leader_election.cc:288] T 00000000000000000000000000000000 P b858c2bd3c2346bf9ac073d925ee5ef1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 04:07:02.537597 31872 external_mini_cluster.cc:1428] Setting key b7808a36b715822c4121421cf869a3f2
I20260810 04:07:02.537647 6072 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b858c2bd3c2346bf9ac073d925ee5ef1 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:07:02.537801 6072 raft_consensus.cc:686] T 00000000000000000000000000000000 P b858c2bd3c2346bf9ac073d925ee5ef1 [term 1 LEADER]: Becoming Leader. State: Replica: b858c2bd3c2346bf9ac073d925ee5ef1, State: Running, Role: LEADER
I20260810 04:07:02.537861 6069 sys_catalog.cc:564] T 00000000000000000000000000000000 P b858c2bd3c2346bf9ac073d925ee5ef1 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:07:02.537895 6072 consensus_queue.cc:237] T 00000000000000000000000000000000 P b858c2bd3c2346bf9ac073d925ee5ef1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b858c2bd3c2346bf9ac073d925ee5ef1" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 35131 } }
I20260810 04:07:02.538282 6073 sys_catalog.cc:455] T 00000000000000000000000000000000 P b858c2bd3c2346bf9ac073d925ee5ef1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b858c2bd3c2346bf9ac073d925ee5ef1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b858c2bd3c2346bf9ac073d925ee5ef1" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 35131 } } }
I20260810 04:07:02.538386 6073 sys_catalog.cc:458] T 00000000000000000000000000000000 P b858c2bd3c2346bf9ac073d925ee5ef1 [sys.catalog]: This master's current role is: LEADER
I20260810 04:07:02.538614 6075 sys_catalog.cc:455] T 00000000000000000000000000000000 P b858c2bd3c2346bf9ac073d925ee5ef1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b858c2bd3c2346bf9ac073d925ee5ef1. Latest consensus state: current_term: 1 leader_uuid: "b858c2bd3c2346bf9ac073d925ee5ef1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b858c2bd3c2346bf9ac073d925ee5ef1" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 35131 } } }
I20260810 04:07:02.538757 6075 sys_catalog.cc:458] T 00000000000000000000000000000000 P b858c2bd3c2346bf9ac073d925ee5ef1 [sys.catalog]: This master's current role is: LEADER
I20260810 04:07:02.538942 6082 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 04:07:02.539547 6082 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 04:07:02.541152 6082 catalog_manager.cc:1347] Generated new cluster ID: 1dd3f94c8dd942a4acce7f7524739ff2
I20260810 04:07:02.541208 6082 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 10 04:07:02 dist-test-slave-8w9v krb5kdc[5977](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334822, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.31.32.62@KRBTEST.COM
I20260810 04:07:02.549660 6078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:02.538261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49934 (local address 127.31.32.62:35131)
0810 04:07:02.538770 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0810 04:07:02.538781 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 04:07:02.538834 (+ 53us) server_negotiation.cc:407] Connection header received
0810 04:07:02.538953 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:07:02.538960 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:07:02.539021 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:07:02.540250 (+ 1229us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:07:02.540329 (+ 79us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:02.541437 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:02.542244 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:02.542731 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:02.542881 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:02.545456 (+ 2575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:07:02.545464 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:07:02.545465 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:07:02.545476 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:07:02.547828 (+ 2352us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:07:02.548369 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:07:02.548373 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:07:02.548373 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:07:02.548509 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:07:02.548733 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:07:02.548734 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:07:02.548734 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:07:02.549005 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:07:02.549193 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0810 04:07:02.549280 (+ 87us) server_negotiation.cc:299] Negotiation successful
0810 04:07:02.549430 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":451,"thread_start_us":80,"threads_started":1}
I20260810 04:07:02.554823 6082 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 04:07:02.555369 6082 catalog_manager.cc:1504] Loading token signing keys...
I20260810 04:07:02.560910 6082 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b858c2bd3c2346bf9ac073d925ee5ef1: Generated new TSK 0
I20260810 04:07:02.561086 6082 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.31.32.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.32.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.32.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/krb5kdc/oryx_127.31.32.1.keytab.
Entry for principal oryx/127.31.32.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/krb5kdc/oryx_127.31.32.1.keytab.
I20260810 04:07:02.599066 31872 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskws89Io/build/release/bin/kudu
/tmp/dist-test-taskws89Io/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.32.1:0
--local_ip_for_outbound_sockets=127.31.32.1
--webserver_interface=127.31.32.1
--webserver_port=0
--tserver_master_addrs=127.31.32.62:35131
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/krb5kdc/oryx_127.31.32.1.keytab
--principal=oryx/127.31.32.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.32.20:35351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:07:02.689004 6100 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:07:02.689209 6100 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:07:02.689235 6100 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:07:02.690392 6100 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:07:02.690433 6100 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:07:02.690459 6100 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.32.1
I20260810 04:07:02.691913 6100 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.32.20:35351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/krb5kdc/oryx_127.31.32.1.keytab
--principal=oryx/127.31.32.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.32.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.32.1
--webserver_port=0
--tserver_master_addrs=127.31.32.62:35131
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6100
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.32.1
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:07:02.692201 6100 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:07:02.692445 6100 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:07:02.697448 6109 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:07:02.697475 6105 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:07:02.697438 6106 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:07:02.697651 6100 server_base.cc:1034] running on GCE node
I20260810 04:07:02.697854 6100 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:07:02.698113 6100 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:07:02.699277 6100 hybrid_clock.cc:648] HybridClock initialized: now 1786334822699261 us; error 35 us; skew 500 ppm
Aug 10 04:07:02 dist-test-slave-8w9v krb5kdc[5977](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334822, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.32.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:07:02.701874 6100 init.cc:377] Logged in from keytab as oryx/127.31.32.1@KRBTEST.COM (short username oryx)
I20260810 04:07:02.702315 6100 webserver.cc:460] Webserver started at http://127.31.32.1:37967/ using document root <none> and password file <none>
I20260810 04:07:02.702584 6100 fs_manager.cc:359] Metadata directory not provided
I20260810 04:07:02.702631 6100 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:07:02.702782 6100 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:07:02.703799 6100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/minicluster-data/ts-0/data/instance:
uuid: "ec8b4c39ed3f419d99b5387ca3e51ca6"
format_stamp: "Formatted at 2026-08-10 04:07:02 on dist-test-slave-8w9v"
server_key: "76da7e23259e206e4fc803fd8467b6ea"
server_key_iv: "3a0efacd3c371a71b556b76e3d72ca7d"
server_key_version: "encryptionkey@0"
I20260810 04:07:02.704222 6100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/minicluster-data/ts-0/wal/instance:
uuid: "ec8b4c39ed3f419d99b5387ca3e51ca6"
format_stamp: "Formatted at 2026-08-10 04:07:02 on dist-test-slave-8w9v"
server_key: "76da7e23259e206e4fc803fd8467b6ea"
server_key_iv: "3a0efacd3c371a71b556b76e3d72ca7d"
server_key_version: "encryptionkey@0"
I20260810 04:07:02.705561 6100 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260810 04:07:02.706282 6117 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:07:02.706463 6100 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 04:07:02.706562 6100 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/minicluster-data/ts-0/data,/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/minicluster-data/ts-0/wal
uuid: "ec8b4c39ed3f419d99b5387ca3e51ca6"
format_stamp: "Formatted at 2026-08-10 04:07:02 on dist-test-slave-8w9v"
server_key: "76da7e23259e206e4fc803fd8467b6ea"
server_key_iv: "3a0efacd3c371a71b556b76e3d72ca7d"
server_key_version: "encryptionkey@0"
I20260810 04:07:02.706646 6100 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:07:02.720896 6100 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:07:02.722672 6100 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:07:02.722834 6100 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:07:02.723050 6100 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:07:02.723363 6100 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:07:02.723417 6100 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:07:02.723457 6100 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:07:02.723485 6100 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:07:02.729324 6100 rpc_server.cc:307] RPC server started. Bound to: 127.31.32.1:46215
I20260810 04:07:02.729368 6238 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.32.1:46215 every 8 connection(s)
I20260810 04:07:02.729729 6100 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/minicluster-data/ts-0/data/info.pb
I20260810 04:07:02.734560 31872 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskws89Io/build/release/bin/kudu as pid 6100
I20260810 04:07:02.734676 31872 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/minicluster-data/ts-0/wal/instance
I20260810 04:07:02.734877 31872 external_mini_cluster.cc:1428] Setting key 5cf054090fb40a4465e229d7ae4d9cc0
Aug 10 04:07:02 dist-test-slave-8w9v krb5kdc[5977](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334822, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.32.1@KRBTEST.COM for oryx/127.31.32.62@KRBTEST.COM
I20260810 04:07:02.739369 6241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:02.730470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:35131 (local address 127.31.32.1:42373)
0810 04:07:02.730643 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0810 04:07:02.730678 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0810 04:07:02.731130 (+ 452us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:07:02.731365 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:07:02.731368 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:07:02.731923 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:07:02.732098 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:02.732103 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:02.732785 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:02.732786 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:02.733543 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:02.733545 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:02.733856 (+ 311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:02.733857 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:02.733948 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:02.734379 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:07:02.734387 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:07:02.736040 (+ 1653us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:07:02.737864 (+ 1824us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:07:02.737872 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:07:02.737875 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:07:02.738184 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:07:02.738414 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:07:02.738415 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:07:02.738415 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:07:02.738576 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:07:02.738853 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:07:02.738858 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:07:02.739009 (+ 151us) client_negotiation.cc:769] Sending connection context
0810 04:07:02.739096 (+ 87us) client_negotiation.cc:240] Negotiation successful
0810 04:07:02.739209 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":91,"threads_started":1}
I20260810 04:07:02.739437 6078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:02.730480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.1:42373 (local address 127.31.32.62:35131)
0810 04:07:02.730535 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0810 04:07:02.730537 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:07:02.730764 (+ 227us) server_negotiation.cc:407] Connection header received
0810 04:07:02.731211 (+ 447us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:07:02.731213 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:07:02.731225 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:07:02.731278 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:07:02.732230 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:02.732711 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:02.733613 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:02.733787 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:02.733836 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:02.736154 (+ 2318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:07:02.736156 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:07:02.736156 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:07:02.736164 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:07:02.737767 (+ 1603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:07:02.738259 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:07:02.738260 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:07:02.738261 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:07:02.738316 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:07:02.738657 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:07:02.738658 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:07:02.738658 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:07:02.738758 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:07:02.738865 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0810 04:07:02.739178 (+ 313us) server_negotiation.cc:299] Negotiation successful
0810 04:07:02.739283 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260810 04:07:02.740051 6239 heartbeater.cc:347] Connected to a master server at 127.31.32.62:35131
I20260810 04:07:02.740198 6239 heartbeater.cc:464] Registering TS with master...
I20260810 04:07:02.740393 6239 heartbeater.cc:511] Master 127.31.32.62:35131 requested a full tablet report, sending...
I20260810 04:07:02.741057 6028 ts_manager.cc:194] Registered new tserver with Master: ec8b4c39ed3f419d99b5387ca3e51ca6 (127.31.32.1:46215)
I20260810 04:07:02.742139 6028 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.31.32.1@KRBTEST.COM'} at 127.31.32.1:42373
WARNING: no policy specified for oryx/127.31.32.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.32.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.32.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/krb5kdc/oryx_127.31.32.2.keytab.
Entry for principal oryx/127.31.32.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/krb5kdc/oryx_127.31.32.2.keytab.
I20260810 04:07:02.762795 31872 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskws89Io/build/release/bin/kudu
/tmp/dist-test-taskws89Io/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.32.2:0
--local_ip_for_outbound_sockets=127.31.32.2
--webserver_interface=127.31.32.2
--webserver_port=0
--tserver_master_addrs=127.31.32.62:35131
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/krb5kdc/oryx_127.31.32.2.keytab
--principal=oryx/127.31.32.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.32.20:35351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:07:02.844666 6245 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:07:02.844889 6245 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:07:02.844930 6245 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:07:02.846050 6245 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:07:02.846110 6245 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:07:02.846161 6245 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.32.2
I20260810 04:07:02.847594 6245 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.32.20:35351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/krb5kdc/oryx_127.31.32.2.keytab
--principal=oryx/127.31.32.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.32.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.32.2
--webserver_port=0
--tserver_master_addrs=127.31.32.62:35131
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6245
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.32.2
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:07:02.847860 6245 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:07:02.848083 6245 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:07:02.852926 6250 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:07:02.852933 6254 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:07:02.852936 6252 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:07:02.853266 6245 server_base.cc:1034] running on GCE node
I20260810 04:07:02.853503 6245 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:07:02.853739 6245 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:07:02.854899 6245 hybrid_clock.cc:648] HybridClock initialized: now 1786334822854896 us; error 41 us; skew 500 ppm
Aug 10 04:07:02 dist-test-slave-8w9v krb5kdc[5977](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334822, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.32.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:07:02.857499 6245 init.cc:377] Logged in from keytab as oryx/127.31.32.2@KRBTEST.COM (short username oryx)
I20260810 04:07:02.857911 6245 webserver.cc:460] Webserver started at http://127.31.32.2:44995/ using document root <none> and password file <none>
I20260810 04:07:02.858144 6245 fs_manager.cc:359] Metadata directory not provided
I20260810 04:07:02.858212 6245 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:07:02.858366 6245 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:07:02.859366 6245 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/minicluster-data/ts-1/data/instance:
uuid: "378cfcc378724919ac8f9f7d19b06bfa"
format_stamp: "Formatted at 2026-08-10 04:07:02 on dist-test-slave-8w9v"
server_key: "79c9be715fbc904f5ab9be749ad5b090"
server_key_iv: "71e4f8f11f6cbef90f36800ecad423d1"
server_key_version: "encryptionkey@0"
I20260810 04:07:02.859723 6245 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/minicluster-data/ts-1/wal/instance:
uuid: "378cfcc378724919ac8f9f7d19b06bfa"
format_stamp: "Formatted at 2026-08-10 04:07:02 on dist-test-slave-8w9v"
server_key: "79c9be715fbc904f5ab9be749ad5b090"
server_key_iv: "71e4f8f11f6cbef90f36800ecad423d1"
server_key_version: "encryptionkey@0"
I20260810 04:07:02.861070 6245 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 04:07:02.861800 6263 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:07:02.861974 6245 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 04:07:02.862075 6245 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/minicluster-data/ts-1/data,/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/minicluster-data/ts-1/wal
uuid: "378cfcc378724919ac8f9f7d19b06bfa"
format_stamp: "Formatted at 2026-08-10 04:07:02 on dist-test-slave-8w9v"
server_key: "79c9be715fbc904f5ab9be749ad5b090"
server_key_iv: "71e4f8f11f6cbef90f36800ecad423d1"
server_key_version: "encryptionkey@0"
I20260810 04:07:02.862144 6245 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:07:02.888535 6245 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:07:02.890426 6245 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:07:02.890592 6245 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:07:02.890827 6245 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:07:02.891147 6245 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:07:02.891198 6245 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:07:02.891240 6245 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:07:02.891268 6245 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:07:02.897279 6245 rpc_server.cc:307] RPC server started. Bound to: 127.31.32.2:41349
I20260810 04:07:02.897346 6381 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.32.2:41349 every 8 connection(s)
I20260810 04:07:02.897732 6245 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/minicluster-data/ts-1/data/info.pb
I20260810 04:07:02.897899 31872 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskws89Io/build/release/bin/kudu as pid 6245
I20260810 04:07:02.898015 31872 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/minicluster-data/ts-1/wal/instance
I20260810 04:07:02.898262 31872 external_mini_cluster.cc:1428] Setting key 53e3945b7596ba657093945eb0ff9aba
Aug 10 04:07:02 dist-test-slave-8w9v krb5kdc[5977](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334822, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.32.2@KRBTEST.COM for oryx/127.31.32.62@KRBTEST.COM
WARNING: no policy specified for oryx/127.31.32.3@KRBTEST.COM; defaulting to no policy
I20260810 04:07:02.909132 6078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:02.898861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.2:32875 (local address 127.31.32.62:35131)
0810 04:07:02.898914 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0810 04:07:02.898916 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:07:02.898927 (+ 11us) server_negotiation.cc:407] Connection header received
0810 04:07:02.899591 (+ 664us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:07:02.899592 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:07:02.899601 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:07:02.899652 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:07:02.901019 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:02.901791 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:02.903217 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:02.903419 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:02.903491 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:02.906189 (+ 2698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:07:02.906191 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:07:02.906191 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:07:02.906200 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:07:02.907713 (+ 1513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:07:02.908171 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:07:02.908172 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:07:02.908173 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:07:02.908238 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:07:02.908504 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:07:02.908505 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:07:02.908506 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:07:02.908635 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:07:02.908717 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0810 04:07:02.908947 (+ 230us) server_negotiation.cc:299] Negotiation successful
0810 04:07:02.909017 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260810 04:07:02.909461 6384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:02.898548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:35131 (local address 127.31.32.2:32875)
0810 04:07:02.898745 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0810 04:07:02.898783 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0810 04:07:02.899483 (+ 700us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:07:02.899728 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:07:02.899732 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:07:02.900625 (+ 893us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:07:02.900917 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:02.900925 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:02.901899 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:02.901899 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:02.903113 (+ 1214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:02.903116 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:02.903525 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:02.903526 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:02.903681 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:02.904330 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:07:02.904349 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:07:02.906080 (+ 1731us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:07:02.907812 (+ 1732us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:07:02.907819 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:07:02.907821 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:07:02.908052 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:07:02.908310 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:07:02.908310 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:07:02.908310 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:07:02.908426 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:07:02.908712 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:07:02.908717 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:07:02.908828 (+ 111us) client_negotiation.cc:769] Sending connection context
0810 04:07:02.908901 (+ 73us) client_negotiation.cc:240] Negotiation successful
0810 04:07:02.909134 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":100,"threads_started":1}
I20260810 04:07:02.909921 6382 heartbeater.cc:347] Connected to a master server at 127.31.32.62:35131
I20260810 04:07:02.910012 6382 heartbeater.cc:464] Registering TS with master...
I20260810 04:07:02.910303 6382 heartbeater.cc:511] Master 127.31.32.62:35131 requested a full tablet report, sending...
I20260810 04:07:02.910773 6028 ts_manager.cc:194] Registered new tserver with Master: 378cfcc378724919ac8f9f7d19b06bfa (127.31.32.2:41349)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.32.3@KRBTEST.COM" created.
I20260810 04:07:02.911199 6028 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.31.32.2@KRBTEST.COM'} at 127.31.32.2:32875
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.32.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/krb5kdc/oryx_127.31.32.3.keytab.
Entry for principal oryx/127.31.32.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/krb5kdc/oryx_127.31.32.3.keytab.
I20260810 04:07:02.926782 31872 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskws89Io/build/release/bin/kudu
/tmp/dist-test-taskws89Io/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.32.3:0
--local_ip_for_outbound_sockets=127.31.32.3
--webserver_interface=127.31.32.3
--webserver_port=0
--tserver_master_addrs=127.31.32.62:35131
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/krb5kdc/oryx_127.31.32.3.keytab
--principal=oryx/127.31.32.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.32.20:35351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:07:03.008435 6388 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:07:03.008649 6388 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:07:03.008675 6388 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:07:03.009958 6388 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:07:03.009999 6388 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:07:03.010046 6388 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.32.3
I20260810 04:07:03.011543 6388 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.32.20:35351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/krb5kdc/oryx_127.31.32.3.keytab
--principal=oryx/127.31.32.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.32.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.32.3
--webserver_port=0
--tserver_master_addrs=127.31.32.62:35131
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6388
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.32.3
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:07:03.011808 6388 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:07:03.012049 6388 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:07:03.017174 6396 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:07:03.017244 6393 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:07:03.017174 6394 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:07:03.017409 6388 server_base.cc:1034] running on GCE node
I20260810 04:07:03.017632 6388 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:07:03.017922 6388 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:07:03.019176 6388 hybrid_clock.cc:648] HybridClock initialized: now 1786334823019159 us; error 27 us; skew 500 ppm
Aug 10 04:07:03 dist-test-slave-8w9v krb5kdc[5977](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334823, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.32.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:07:03.021766 6388 init.cc:377] Logged in from keytab as oryx/127.31.32.3@KRBTEST.COM (short username oryx)
I20260810 04:07:03.022306 6388 webserver.cc:460] Webserver started at http://127.31.32.3:35853/ using document root <none> and password file <none>
I20260810 04:07:03.022559 6388 fs_manager.cc:359] Metadata directory not provided
I20260810 04:07:03.022629 6388 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:07:03.022778 6388 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:07:03.023836 6388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/minicluster-data/ts-2/data/instance:
uuid: "71c25e47472544fa9e042cfcc7bf4f96"
format_stamp: "Formatted at 2026-08-10 04:07:03 on dist-test-slave-8w9v"
server_key: "6e5f63a4fc0713ed71eeebb6402a14ab"
server_key_iv: "509ed22b9999d7360da495e7fed07250"
server_key_version: "encryptionkey@0"
I20260810 04:07:03.024302 6388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/minicluster-data/ts-2/wal/instance:
uuid: "71c25e47472544fa9e042cfcc7bf4f96"
format_stamp: "Formatted at 2026-08-10 04:07:03 on dist-test-slave-8w9v"
server_key: "6e5f63a4fc0713ed71eeebb6402a14ab"
server_key_iv: "509ed22b9999d7360da495e7fed07250"
server_key_version: "encryptionkey@0"
I20260810 04:07:03.025709 6388 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260810 04:07:03.026484 6405 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:07:03.026713 6388 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 04:07:03.026821 6388 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/minicluster-data/ts-2/data,/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/minicluster-data/ts-2/wal
uuid: "71c25e47472544fa9e042cfcc7bf4f96"
format_stamp: "Formatted at 2026-08-10 04:07:03 on dist-test-slave-8w9v"
server_key: "6e5f63a4fc0713ed71eeebb6402a14ab"
server_key_iv: "509ed22b9999d7360da495e7fed07250"
server_key_version: "encryptionkey@0"
I20260810 04:07:03.026881 6388 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:07:03.037309 6388 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:07:03.040021 6388 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:07:03.040194 6388 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:07:03.040398 6388 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:07:03.040660 6388 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:07:03.040694 6388 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:07:03.040717 6388 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:07:03.040738 6388 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:07:03.046957 6388 rpc_server.cc:307] RPC server started. Bound to: 127.31.32.3:32929
I20260810 04:07:03.047039 6537 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.32.3:32929 every 8 connection(s)
I20260810 04:07:03.047354 6388 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/minicluster-data/ts-2/data/info.pb
I20260810 04:07:03.051577 31872 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskws89Io/build/release/bin/kudu as pid 6388
I20260810 04:07:03.051682 31872 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/minicluster-data/ts-2/wal/instance
I20260810 04:07:03.051887 31872 external_mini_cluster.cc:1428] Setting key 4475498ed62d39c75bc4c19c6a003e81
Aug 10 04:07:03 dist-test-slave-8w9v krb5kdc[5977](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334823, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.32.3@KRBTEST.COM for oryx/127.31.32.62@KRBTEST.COM
I20260810 04:07:03.057472 6078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:03.048080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.3:43917 (local address 127.31.32.62:35131)
0810 04:07:03.048155 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0810 04:07:03.048157 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:07:03.048397 (+ 240us) server_negotiation.cc:407] Connection header received
0810 04:07:03.048960 (+ 563us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:07:03.048962 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:07:03.048977 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:07:03.049049 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:07:03.050240 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:03.050776 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:03.052087 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:03.052445 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:03.052526 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:03.054629 (+ 2103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:07:03.054631 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:07:03.054631 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:07:03.054639 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:07:03.056289 (+ 1650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:07:03.056654 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:07:03.056655 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:07:03.056655 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:07:03.056731 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:07:03.056962 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:07:03.056962 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:07:03.056962 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:07:03.057072 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:07:03.057135 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0810 04:07:03.057333 (+ 198us) server_negotiation.cc:299] Negotiation successful
0810 04:07:03.057406 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260810 04:07:03.057665 6540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:03.048047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:35131 (local address 127.31.32.3:43917)
0810 04:07:03.048274 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0810 04:07:03.048310 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0810 04:07:03.048847 (+ 537us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:07:03.049123 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:07:03.049127 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:07:03.049928 (+ 801us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:07:03.050128 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:03.050134 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:03.050896 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:03.050897 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:03.052002 (+ 1105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:03.052006 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:03.052549 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:03.052550 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:03.052704 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:03.053185 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:07:03.053193 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:07:03.054540 (+ 1347us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:07:03.056371 (+ 1831us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:07:03.056377 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:07:03.056378 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:07:03.056584 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:07:03.056799 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:07:03.056799 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:07:03.056799 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:07:03.056904 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:07:03.057125 (+ 221us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:07:03.057127 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:07:03.057231 (+ 104us) client_negotiation.cc:769] Sending connection context
0810 04:07:03.057306 (+ 75us) client_negotiation.cc:240] Negotiation successful
0810 04:07:03.057431 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":128,"threads_started":1}
I20260810 04:07:03.058239 6538 heartbeater.cc:347] Connected to a master server at 127.31.32.62:35131
I20260810 04:07:03.058324 6538 heartbeater.cc:464] Registering TS with master...
I20260810 04:07:03.058544 6538 heartbeater.cc:511] Master 127.31.32.62:35131 requested a full tablet report, sending...
I20260810 04:07:03.058957 6028 ts_manager.cc:194] Registered new tserver with Master: 71c25e47472544fa9e042cfcc7bf4f96 (127.31.32.3:32929)
I20260810 04:07:03.059307 6028 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.31.32.3@KRBTEST.COM'} at 127.31.32.3:43917
I20260810 04:07:03.063854 31872 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.31.32.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.32.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.32.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/krb5kdc/oryx_127.31.32.61.keytab.
Entry for principal oryx/127.31.32.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/krb5kdc/oryx_127.31.32.61.keytab.
I20260810 04:07:03.091174 31872 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskws89Io/build/release/bin/kudu
/tmp/dist-test-taskws89Io/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.32.61:40815
--webserver_interface=127.31.32.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/krb5kdc/oryx_127.31.32.61.keytab
--principal=oryx/127.31.32.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.32.20:35351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.32.62:35131,127.31.32.61:40815
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:07:03.175508 6543 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:07:03.175715 6543 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:07:03.175740 6543 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:07:03.176887 6543 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:07:03.176928 6543 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:07:03.176944 6543 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:07:03.176959 6543 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:07:03.176973 6543 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:07:03.178574 6543 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.32.20:35351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.31.32.62:35131,127.31.32.61:40815
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/krb5kdc/oryx_127.31.32.61.keytab
--principal=oryx/127.31.32.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.32.61:40815
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.31.32.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6543
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
W20260810 04:07:03.178692 6543 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.31.32.62:35131,127.31.32.61:40815'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260810 04:07:03.178807 6543 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:07:03.178997 6543 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:07:03.183962 6551 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:07:03.183938 6553 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:07:03.184010 6550 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:07:03.184188 6543 server_base.cc:1034] running on GCE node
I20260810 04:07:03.184432 6543 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:07:03.184707 6543 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:07:03.185880 6543 hybrid_clock.cc:648] HybridClock initialized: now 1786334823185868 us; error 49 us; skew 500 ppm
Aug 10 04:07:03 dist-test-slave-8w9v krb5kdc[5977](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334823, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.32.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:07:03.188417 6543 init.cc:377] Logged in from keytab as oryx/127.31.32.61@KRBTEST.COM (short username oryx)
I20260810 04:07:03.188877 6543 webserver.cc:460] Webserver started at http://127.31.32.61:39063/ using document root <none> and password file <none>
I20260810 04:07:03.189078 6543 fs_manager.cc:359] Metadata directory not provided
I20260810 04:07:03.189119 6543 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:07:03.189227 6543 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:07:03.190236 6543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/minicluster-data/master-1/data/instance:
uuid: "2138c4c9465e4cd19eeef783497f22e5"
format_stamp: "Formatted at 2026-08-10 04:07:03 on dist-test-slave-8w9v"
server_key: "0c8529fcbf14bc0ad04044de71d8a945"
server_key_iv: "d595724c66aeb3d6272f5f79f0175cf9"
server_key_version: "encryptionkey@0"
I20260810 04:07:03.190544 6543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/minicluster-data/master-1/wal/instance:
uuid: "2138c4c9465e4cd19eeef783497f22e5"
format_stamp: "Formatted at 2026-08-10 04:07:03 on dist-test-slave-8w9v"
server_key: "0c8529fcbf14bc0ad04044de71d8a945"
server_key_iv: "d595724c66aeb3d6272f5f79f0175cf9"
server_key_version: "encryptionkey@0"
I20260810 04:07:03.191848 6543 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 04:07:03.192631 6560 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:07:03.192807 6543 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260810 04:07:03.192902 6543 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/minicluster-data/master-1/data,/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/minicluster-data/master-1/wal
uuid: "2138c4c9465e4cd19eeef783497f22e5"
format_stamp: "Formatted at 2026-08-10 04:07:03 on dist-test-slave-8w9v"
server_key: "0c8529fcbf14bc0ad04044de71d8a945"
server_key_iv: "d595724c66aeb3d6272f5f79f0175cf9"
server_key_version: "encryptionkey@0"
I20260810 04:07:03.192955 6543 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:07:03.216610 6543 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:07:03.218395 6543 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:07:03.218508 6543 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:07:03.222299 6543 rpc_server.cc:307] RPC server started. Bound to: 127.31.32.61:40815
I20260810 04:07:03.222366 6618 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.32.61:40815 every 8 connection(s)
I20260810 04:07:03.222795 6543 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/minicluster-data/master-1/data/info.pb
I20260810 04:07:03.223809 6620 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:07:03.225373 6620 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.32.62" port: 35131 } has no permanent_uuid. Determining permanent_uuid...
I20260810 04:07:03.226002 31872 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskws89Io/build/release/bin/kudu as pid 6543
Aug 10 04:07:03 dist-test-slave-8w9v krb5kdc[5977](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334823, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.32.61@KRBTEST.COM for oryx/127.31.32.62@KRBTEST.COM
Aug 10 04:07:03 dist-test-slave-8w9v krb5kdc[5977](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334822, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.31.32.61@KRBTEST.COM
I20260810 04:07:03.235268 6078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:03.226132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49940 (local address 127.31.32.62:35131)
0810 04:07:03.226197 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0810 04:07:03.226199 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:07:03.226482 (+ 283us) server_negotiation.cc:407] Connection header received
0810 04:07:03.227150 (+ 668us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:07:03.227151 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:07:03.227164 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:07:03.227226 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:07:03.227761 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:03.228381 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:03.229427 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:03.229755 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:03.229819 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:03.232198 (+ 2379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:07:03.232200 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:07:03.232200 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:07:03.232207 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:07:03.233760 (+ 1553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:07:03.234248 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:07:03.234249 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:07:03.234249 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:07:03.234299 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:07:03.234584 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:07:03.234584 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:07:03.234584 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:07:03.234723 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:07:03.234776 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0810 04:07:03.235085 (+ 309us) server_negotiation.cc:299] Negotiation successful
0810 04:07:03.235170 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260810 04:07:03.235472 6622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:03.226112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:35131 (local address 127.0.0.1:49940)
0810 04:07:03.226316 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0810 04:07:03.226383 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0810 04:07:03.227020 (+ 637us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:07:03.227265 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:07:03.227268 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:07:03.227518 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:07:03.227663 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:03.227668 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:03.228477 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:03.228478 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:03.229350 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:03.229352 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:03.229830 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:03.229831 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:03.229967 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:03.230510 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:07:03.230523 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:07:03.232062 (+ 1539us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:07:03.233845 (+ 1783us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:07:03.233852 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:07:03.233855 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:07:03.234170 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:07:03.234382 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:07:03.234383 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:07:03.234383 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:07:03.234513 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:07:03.234768 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:07:03.234773 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:07:03.234931 (+ 158us) client_negotiation.cc:769] Sending connection context
0810 04:07:03.235044 (+ 113us) client_negotiation.cc:240] Negotiation successful
0810 04:07:03.235201 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":98,"threads_started":1}
I20260810 04:07:03.236212 6620 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.32.61" port: 40815 } has no permanent_uuid. Determining permanent_uuid...
I20260810 04:07:03.236428 6624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:03.226549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48534 (local address 127.31.32.61:40815)
0810 04:07:03.226683 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0810 04:07:03.226688 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 04:07:03.226725 (+ 37us) server_negotiation.cc:407] Connection header received
0810 04:07:03.227022 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:07:03.227031 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:07:03.227058 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:07:03.227146 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:07:03.227765 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:03.228925 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:03.229804 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:03.230244 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:03.230304 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:03.232758 (+ 2454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:07:03.232765 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:07:03.232765 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:07:03.232772 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:07:03.234648 (+ 1876us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:07:03.235048 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:07:03.235052 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:07:03.235052 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:07:03.235117 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:07:03.235549 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:07:03.235550 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:07:03.235550 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:07:03.235734 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:07:03.235915 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0810 04:07:03.236062 (+ 147us) server_negotiation.cc:299] Negotiation successful
0810 04:07:03.236165 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":80,"threads_started":1}
Aug 10 04:07:03 dist-test-slave-8w9v krb5kdc[5977](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334823, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.32.61@KRBTEST.COM for oryx/127.31.32.61@KRBTEST.COM
I20260810 04:07:03.242419 6622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:03.236506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.61:40815 (local address 127.0.0.1:48536)
0810 04:07:03.236647 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0810 04:07:03.236657 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 04:07:03.236708 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:07:03.236835 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:07:03.236836 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:07:03.236988 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:07:03.237018 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:03.237020 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:03.237703 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:03.237703 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:03.238176 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:03.238177 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:03.238576 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:03.238577 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:03.238658 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:03.239081 (+ 423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:07:03.239086 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:07:03.240401 (+ 1315us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:07:03.241641 (+ 1240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:07:03.241642 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:07:03.241642 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:07:03.241798 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:07:03.241968 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:07:03.241968 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:07:03.241969 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:07:03.242017 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:07:03.242207 (+ 190us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:07:03.242208 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:07:03.242239 (+ 31us) client_negotiation.cc:769] Sending connection context
0810 04:07:03.242288 (+ 49us) client_negotiation.cc:240] Negotiation successful
0810 04:07:03.242360 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126}
I20260810 04:07:03.242532 6624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:03.236568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48536 (local address 127.31.32.61:40815)
0810 04:07:03.236608 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0810 04:07:03.236610 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:07:03.236695 (+ 85us) server_negotiation.cc:407] Connection header received
0810 04:07:03.236761 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:07:03.236762 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:07:03.236771 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:07:03.236831 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:07:03.237103 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:03.237580 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:03.238272 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:03.238501 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:03.238579 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:03.240478 (+ 1899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:07:03.240480 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:07:03.240480 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:07:03.240486 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:07:03.241572 (+ 1086us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:07:03.241863 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:07:03.241864 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:07:03.241864 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:07:03.241911 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:07:03.242069 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:07:03.242070 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:07:03.242070 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:07:03.242156 (+ 86us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:07:03.242202 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0810 04:07:03.242301 (+ 99us) server_negotiation.cc:299] Negotiation successful
0810 04:07:03.242404 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":46,"server-negotiator.queue_time_us":22}
I20260810 04:07:03.243677 6620 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2138c4c9465e4cd19eeef783497f22e5: Bootstrap starting.
I20260810 04:07:03.244626 6620 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2138c4c9465e4cd19eeef783497f22e5: Neither blocks nor log segments found. Creating new log.
I20260810 04:07:03.244861 6620 log.cc:824] T 00000000000000000000000000000000 P 2138c4c9465e4cd19eeef783497f22e5: Log is configured to *not* fsync() on all Append() calls
I20260810 04:07:03.245548 6620 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2138c4c9465e4cd19eeef783497f22e5: No bootstrap required, opened a new log
I20260810 04:07:03.246884 6620 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2138c4c9465e4cd19eeef783497f22e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b858c2bd3c2346bf9ac073d925ee5ef1" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 35131 } } peers { permanent_uuid: "2138c4c9465e4cd19eeef783497f22e5" member_type: VOTER last_known_addr { host: "127.31.32.61" port: 40815 } }
I20260810 04:07:03.247002 6620 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2138c4c9465e4cd19eeef783497f22e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:07:03.247025 6620 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2138c4c9465e4cd19eeef783497f22e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2138c4c9465e4cd19eeef783497f22e5, State: Initialized, Role: FOLLOWER
I20260810 04:07:03.247148 6620 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2138c4c9465e4cd19eeef783497f22e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b858c2bd3c2346bf9ac073d925ee5ef1" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 35131 } } peers { permanent_uuid: "2138c4c9465e4cd19eeef783497f22e5" member_type: VOTER last_known_addr { host: "127.31.32.61" port: 40815 } }
I20260810 04:07:03.247424 6626 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2138c4c9465e4cd19eeef783497f22e5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b858c2bd3c2346bf9ac073d925ee5ef1" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 35131 } } peers { permanent_uuid: "2138c4c9465e4cd19eeef783497f22e5" member_type: VOTER last_known_addr { host: "127.31.32.61" port: 40815 } } }
I20260810 04:07:03.247498 6620 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2138c4c9465e4cd19eeef783497f22e5 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:07:03.247519 6626 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2138c4c9465e4cd19eeef783497f22e5 [sys.catalog]: This master's current role is: FOLLOWER
W20260810 04:07:03.249223 6637 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2138c4c9465e4cd19eeef783497f22e5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 04:07:03.249286 6637 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 04:07:03.251065 6543 master_runner.cc:428] Detected that this master 2138c4c9465e4cd19eeef783497f22e5 is joining an existing cluster
I20260810 04:07:03.251130 6543 master_runner.cc:432] Initiating AddMaster RPC to add 127.31.32.61:40815
I20260810 04:07:03.259178 6078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:03.252245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49952 (local address 127.31.32.62:35131)
0810 04:07:03.252300 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0810 04:07:03.252303 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:07:03.252370 (+ 67us) server_negotiation.cc:407] Connection header received
0810 04:07:03.252617 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:07:03.252619 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:07:03.252631 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:07:03.252660 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:07:03.253090 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:03.253575 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:03.254729 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:03.255033 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:03.255111 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:03.256231 (+ 1120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:07:03.256233 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:07:03.256233 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:07:03.256240 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:07:03.257938 (+ 1698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:07:03.258309 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:07:03.258310 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:07:03.258311 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:07:03.258381 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:07:03.258597 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:07:03.258598 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:07:03.258598 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:07:03.258760 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:07:03.258829 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0810 04:07:03.258992 (+ 163us) server_negotiation.cc:299] Negotiation successful
0810 04:07:03.259075 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260810 04:07:03.259248 6624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:03.252399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48546 (local address 127.31.32.61:40815)
0810 04:07:03.252442 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0810 04:07:03.252443 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 04:07:03.252449 (+ 6us) server_negotiation.cc:407] Connection header received
0810 04:07:03.252471 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:07:03.252472 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:07:03.252478 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:07:03.252539 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:07:03.252981 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:03.253708 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:03.255167 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:03.255459 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:03.255523 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:03.256516 (+ 993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:07:03.256518 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:07:03.256518 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:07:03.256525 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:07:03.257968 (+ 1443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:07:03.258425 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:07:03.258426 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:07:03.258427 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:07:03.258489 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:07:03.258694 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:07:03.258695 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:07:03.258695 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:07:03.258830 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:07:03.258890 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0810 04:07:03.259028 (+ 138us) server_negotiation.cc:299] Negotiation successful
0810 04:07:03.259127 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":28}
I20260810 04:07:03.259418 6645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:03.252136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.61:40815 (local address 127.0.0.1:48546)
0810 04:07:03.252301 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0810 04:07:03.252313 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 04:07:03.252372 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:07:03.252543 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:07:03.252543 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:07:03.252794 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:07:03.252893 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:03.252895 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:03.253788 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:03.253789 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:03.254635 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:03.254638 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:03.255567 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:03.255568 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:03.255624 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:03.256037 (+ 413us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:07:03.256042 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:07:03.256441 (+ 399us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:07:03.258073 (+ 1632us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:07:03.258074 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:07:03.258075 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:07:03.258340 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:07:03.258554 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:07:03.258554 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:07:03.258555 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:07:03.258625 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:07:03.258883 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:07:03.258884 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:07:03.258922 (+ 38us) client_negotiation.cc:769] Sending connection context
0810 04:07:03.259056 (+ 134us) client_negotiation.cc:240] Negotiation successful
0810 04:07:03.259185 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"mutex_wait_us":51,"thread_start_us":82,"threads_started":1}
I20260810 04:07:03.259907 6644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:03.252116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:35131 (local address 127.0.0.1:49952)
0810 04:07:03.252300 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0810 04:07:03.252313 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 04:07:03.252395 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:07:03.252733 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:07:03.252733 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:07:03.252949 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:07:03.253006 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:03.253008 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:03.253656 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:03.253657 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:03.254636 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:03.254638 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:03.255124 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:03.255124 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:03.255180 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:03.255641 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:07:03.255647 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:07:03.256100 (+ 453us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:07:03.258026 (+ 1926us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:07:03.258027 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:07:03.258027 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:07:03.258236 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:07:03.258435 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:07:03.258435 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:07:03.258436 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:07:03.258526 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:07:03.258837 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:07:03.258838 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:07:03.258882 (+ 44us) client_negotiation.cc:769] Sending connection context
0810 04:07:03.258960 (+ 78us) client_negotiation.cc:240] Negotiation successful
0810 04:07:03.259201 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"mutex_wait_us":20,"thread_start_us":75,"threads_started":1}
Aug 10 04:07:03 dist-test-slave-8w9v krb5kdc[5977](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334822, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.32.62@KRBTEST.COM for oryx/127.31.32.61@KRBTEST.COM
I20260810 04:07:03.269246 6624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:03.262204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48550 (local address 127.31.32.61:40815)
0810 04:07:03.262243 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0810 04:07:03.262245 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:07:03.262382 (+ 137us) server_negotiation.cc:407] Connection header received
0810 04:07:03.262475 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:07:03.262476 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:07:03.262482 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:07:03.262523 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:07:03.263001 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:03.263423 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:03.264522 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:03.264689 (+ 167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:03.264748 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:03.266764 (+ 2016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:07:03.266765 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:07:03.266766 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:07:03.266773 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:07:03.268053 (+ 1280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:07:03.268449 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:07:03.268450 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:07:03.268450 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:07:03.268505 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:07:03.268704 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:07:03.268705 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:07:03.268705 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:07:03.268824 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:07:03.268895 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0810 04:07:03.269080 (+ 185us) server_negotiation.cc:299] Negotiation successful
0810 04:07:03.269155 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260810 04:07:03.269287 6646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:03.262123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.61:40815 (local address 127.0.0.1:48550)
0810 04:07:03.262269 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0810 04:07:03.262317 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0810 04:07:03.262402 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:07:03.262538 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:07:03.262541 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:07:03.262864 (+ 323us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:07:03.262930 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:03.262932 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:03.263498 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:03.263499 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:03.264448 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:03.264451 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:03.264769 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:03.264770 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:03.264864 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:03.265265 (+ 401us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:07:03.265270 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:07:03.266679 (+ 1409us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:07:03.268161 (+ 1482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:07:03.268169 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:07:03.268171 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:07:03.268374 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:07:03.268573 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:07:03.268574 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:07:03.268574 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:07:03.268639 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:07:03.268900 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:07:03.268907 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:07:03.268958 (+ 51us) client_negotiation.cc:769] Sending connection context
0810 04:07:03.269054 (+ 96us) client_negotiation.cc:240] Negotiation successful
0810 04:07:03.269170 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":86,"threads_started":1}
I20260810 04:07:03.269790 6028 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.31.32.61:40815: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "2138c4c9465e4cd19eeef783497f22e5"
member_type: NON_VOTER
last_known_addr {
host: "127.31.32.61"
port: 40815
}
attrs {
promote: true
}
}
dest_uuid: "b858c2bd3c2346bf9ac073d925ee5ef1"
cas_config_opid_index: -1
I20260810 04:07:03.270159 6028 consensus_queue.cc:237] T 00000000000000000000000000000000 P b858c2bd3c2346bf9ac073d925ee5ef1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b858c2bd3c2346bf9ac073d925ee5ef1" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 35131 } } peers { permanent_uuid: "2138c4c9465e4cd19eeef783497f22e5" member_type: NON_VOTER last_known_addr { host: "127.31.32.61" port: 40815 } attrs { promote: true } }
I20260810 04:07:03.270681 6648 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b858c2bd3c2346bf9ac073d925ee5ef1 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 2138c4c9465e4cd19eeef783497f22e5 (127.31.32.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b858c2bd3c2346bf9ac073d925ee5ef1" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 35131 } } peers { permanent_uuid: "2138c4c9465e4cd19eeef783497f22e5" member_type: NON_VOTER last_known_addr { host: "127.31.32.61" port: 40815 } attrs { promote: true } } }
I20260810 04:07:03.270951 6590 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2138c4c9465e4cd19eeef783497f22e5 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:07:03.271844 6590 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2138c4c9465e4cd19eeef783497f22e5 [term 1 FOLLOWER]: Refusing update from remote peer b858c2bd3c2346bf9ac073d925ee5ef1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260810 04:07:03.271884 6648 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.31.32.61:40815
I20260810 04:07:03.272294 6543 master.cc:537] Master@127.31.32.61:40815 shutting down...
I20260810 04:07:03.272295 6648 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b858c2bd3c2346bf9ac073d925ee5ef1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2138c4c9465e4cd19eeef783497f22e5" member_type: NON_VOTER last_known_addr { host: "127.31.32.61" port: 40815 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:07:03.272776 6649 sys_catalog.cc:455] T 00000000000000000000000000000000 P b858c2bd3c2346bf9ac073d925ee5ef1 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "b858c2bd3c2346bf9ac073d925ee5ef1" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b858c2bd3c2346bf9ac073d925ee5ef1" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 35131 } } peers { permanent_uuid: "2138c4c9465e4cd19eeef783497f22e5" member_type: NON_VOTER last_known_addr { host: "127.31.32.61" port: 40815 } attrs { promote: true } } }
I20260810 04:07:03.272866 6649 sys_catalog.cc:458] T 00000000000000000000000000000000 P b858c2bd3c2346bf9ac073d925ee5ef1 [sys.catalog]: This master's current role is: LEADER
W20260810 04:07:03.273356 6013 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260810 04:07:03.273797 6013 consensus_peers.cc:487] T 00000000000000000000000000000000 P b858c2bd3c2346bf9ac073d925ee5ef1 -> Peer 2138c4c9465e4cd19eeef783497f22e5 (127.31.32.61:40815): Couldn't send request to peer 2138c4c9465e4cd19eeef783497f22e5. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260810 04:07:03.275456 6543 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 2138c4c9465e4cd19eeef783497f22e5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 04:07:03.275542 6543 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 2138c4c9465e4cd19eeef783497f22e5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:07:03.275611 6543 tablet_replica.cc:330] T 00000000000000000000000000000000 P 2138c4c9465e4cd19eeef783497f22e5: stopping tablet replica
I20260810 04:07:03.289268 6543 master.cc:552] Master@127.31.32.61:40815 shutdown complete.
I20260810 04:07:03.289326 6543 master_runner.cc:305] Clearing existing system tablet
I20260810 04:07:03.289476 6543 fs_manager.cc:359] Metadata directory not provided
I20260810 04:07:03.289525 6543 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:07:03.290233 6543 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.002s sys 0.000s
I20260810 04:07:03.290733 6654 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:07:03.290887 6543 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:07:03.290949 6543 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260810 04:07:03.290975 6543 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/minicluster-data/master-1/data,/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/minicluster-data/master-1/wal
uuid: "2138c4c9465e4cd19eeef783497f22e5"
format_stamp: "Formatted at 2026-08-10 04:07:03 on dist-test-slave-8w9v"
server_key: "0c8529fcbf14bc0ad04044de71d8a945"
server_key_iv: "d595724c66aeb3d6272f5f79f0175cf9"
server_key_version: "encryptionkey@0"
I20260810 04:07:03.291838 6543 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 2138c4c9465e4cd19eeef783497f22e5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:07:03.293615 6543 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 2138c4c9465e4cd19eeef783497f22e5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260810 04:07:03.293689 6543 log.cc:1196] T 00000000000000000000000000000000 P 2138c4c9465e4cd19eeef783497f22e5: Deleting WAL directory at /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260810 04:07:03.293884 6543 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 2138c4c9465e4cd19eeef783497f22e5: Deleting consensus metadata
I20260810 04:07:03.294016 6543 master_runner.cc:315] Copying system tablet from 127.31.32.62:35131
I20260810 04:07:03.294651 6543 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 2138c4c9465e4cd19eeef783497f22e5: tablet copy: Beginning tablet copy session from remote peer at address 127.31.32.62:35131
I20260810 04:07:03.301283 6078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:03.295108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49966 (local address 127.31.32.62:35131)
0810 04:07:03.295163 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0810 04:07:03.295166 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:07:03.295240 (+ 74us) server_negotiation.cc:407] Connection header received
0810 04:07:03.295294 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:07:03.295295 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:07:03.295306 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:07:03.295361 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:07:03.295716 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:03.296308 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:03.297442 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:03.297626 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:03.297687 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:03.298695 (+ 1008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:07:03.298697 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:07:03.298697 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:07:03.298703 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:07:03.300262 (+ 1559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:07:03.300630 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:07:03.300631 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:07:03.300631 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:07:03.300682 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:07:03.300865 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:07:03.300866 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:07:03.300866 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:07:03.300963 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:07:03.301013 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0810 04:07:03.301166 (+ 153us) server_negotiation.cc:299] Negotiation successful
0810 04:07:03.301223 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260810 04:07:03.301373 6660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:03.295009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:35131 (local address 127.0.0.1:49966)
0810 04:07:03.295147 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0810 04:07:03.295160 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 04:07:03.295231 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:07:03.295368 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:07:03.295369 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:07:03.295566 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:07:03.295635 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:03.295637 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:03.296392 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:03.296394 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:03.297323 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:03.297326 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:03.297704 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:03.297705 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:03.297775 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:03.298195 (+ 420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:07:03.298200 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:07:03.298596 (+ 396us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:07:03.300340 (+ 1744us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:07:03.300341 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:07:03.300341 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:07:03.300561 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:07:03.300741 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:07:03.300742 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:07:03.300742 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:07:03.300805 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:07:03.301025 (+ 220us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:07:03.301026 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:07:03.301072 (+ 46us) client_negotiation.cc:769] Sending connection context
0810 04:07:03.301151 (+ 79us) client_negotiation.cc:240] Negotiation successful
0810 04:07:03.301252 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":94,"threads_started":1}
I20260810 04:07:03.301995 6051 tablet_copy_service.cc:140] P b858c2bd3c2346bf9ac073d925ee5ef1: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 2138c4c9465e4cd19eeef783497f22e5 ({username='oryx', principal='oryx/127.31.32.61@KRBTEST.COM'} at 127.0.0.1:49966)
I20260810 04:07:03.302137 6051 tablet_copy_service.cc:161] P b858c2bd3c2346bf9ac073d925ee5ef1: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 2138c4c9465e4cd19eeef783497f22e5 at {username='oryx', principal='oryx/127.31.32.61@KRBTEST.COM'} at 127.0.0.1:49966: session id = 2138c4c9465e4cd19eeef783497f22e5-00000000000000000000000000000000
I20260810 04:07:03.303076 6051 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P b858c2bd3c2346bf9ac073d925ee5ef1: Tablet Copy: opened 0 blocks and 1 log segments
I20260810 04:07:03.303725 6543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260810 04:07:03.305058 6543 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 2138c4c9465e4cd19eeef783497f22e5: tablet copy: Starting download of 0 data blocks...
I20260810 04:07:03.305203 6543 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 2138c4c9465e4cd19eeef783497f22e5: tablet copy: Starting download of 1 WAL segments...
I20260810 04:07:03.306182 6543 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 2138c4c9465e4cd19eeef783497f22e5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260810 04:07:03.307104 6051 tablet_copy_service.cc:342] P b858c2bd3c2346bf9ac073d925ee5ef1: Request end of tablet copy session 2138c4c9465e4cd19eeef783497f22e5-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.31.32.61@KRBTEST.COM'} at 127.0.0.1:49966
I20260810 04:07:03.307163 6051 tablet_copy_service.cc:433] P b858c2bd3c2346bf9ac073d925ee5ef1: ending tablet copy session 2138c4c9465e4cd19eeef783497f22e5-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 2138c4c9465e4cd19eeef783497f22e5
I20260810 04:07:03.309870 6543 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:07:03.311015 6662 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:07:03.311085 6661 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:07:03.313042 6664 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:07:03.313066 6543 server_base.cc:1034] running on GCE node
I20260810 04:07:03.313239 6543 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:07:03.313410 6543 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:07:03.314530 6543 hybrid_clock.cc:648] HybridClock initialized: now 1786334823314522 us; error 25 us; skew 500 ppm
Aug 10 04:07:03 dist-test-slave-8w9v krb5kdc[5977](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334823, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.32.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:07:03.315887 6543 init.cc:377] Logged in from keytab as oryx/127.31.32.61@KRBTEST.COM (short username oryx)
I20260810 04:07:03.316202 6543 webserver.cc:460] Webserver started at http://127.31.32.61:32917/ using document root <none> and password file <none>
I20260810 04:07:03.316342 6543 fs_manager.cc:359] Metadata directory not provided
I20260810 04:07:03.316380 6543 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:07:03.316998 6543 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260810 04:07:03.317539 6672 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:07:03.317677 6543 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 04:07:03.317726 6543 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/minicluster-data/master-1/data,/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/minicluster-data/master-1/wal
uuid: "2138c4c9465e4cd19eeef783497f22e5"
format_stamp: "Formatted at 2026-08-10 04:07:03 on dist-test-slave-8w9v"
server_key: "0c8529fcbf14bc0ad04044de71d8a945"
server_key_iv: "d595724c66aeb3d6272f5f79f0175cf9"
server_key_version: "encryptionkey@0"
I20260810 04:07:03.317783 6543 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:07:03.324597 6543 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:07:03.326277 6543 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:07:03.329523 6543 rpc_server.cc:307] RPC server started. Bound to: 127.31.32.61:40815
I20260810 04:07:03.329731 6728 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.32.61:40815 every 8 connection(s)
I20260810 04:07:03.329916 6543 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/minicluster-data/master-1/data/info.pb
I20260810 04:07:03.330893 6729 sys_catalog.cc:263] Verifying existing consensus state
I20260810 04:07:03.331446 6729 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2138c4c9465e4cd19eeef783497f22e5: Bootstrap starting.
I20260810 04:07:03.334669 6729 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2138c4c9465e4cd19eeef783497f22e5: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 04:07:03.334923 6729 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2138c4c9465e4cd19eeef783497f22e5: Bootstrap complete.
I20260810 04:07:03.335121 6729 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2138c4c9465e4cd19eeef783497f22e5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b858c2bd3c2346bf9ac073d925ee5ef1" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 35131 } } peers { permanent_uuid: "2138c4c9465e4cd19eeef783497f22e5" member_type: NON_VOTER last_known_addr { host: "127.31.32.61" port: 40815 } attrs { promote: true } }
I20260810 04:07:03.335213 6729 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2138c4c9465e4cd19eeef783497f22e5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2138c4c9465e4cd19eeef783497f22e5, State: Initialized, Role: LEARNER
I20260810 04:07:03.335253 6729 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2138c4c9465e4cd19eeef783497f22e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b858c2bd3c2346bf9ac073d925ee5ef1" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 35131 } } peers { permanent_uuid: "2138c4c9465e4cd19eeef783497f22e5" member_type: NON_VOTER last_known_addr { host: "127.31.32.61" port: 40815 } attrs { promote: true } }
I20260810 04:07:03.335433 6729 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2138c4c9465e4cd19eeef783497f22e5 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:07:03.335448 6732 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2138c4c9465e4cd19eeef783497f22e5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b858c2bd3c2346bf9ac073d925ee5ef1" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 35131 } } peers { permanent_uuid: "2138c4c9465e4cd19eeef783497f22e5" member_type: NON_VOTER last_known_addr { host: "127.31.32.61" port: 40815 } attrs { promote: true } } }
I20260810 04:07:03.335592 6732 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2138c4c9465e4cd19eeef783497f22e5 [sys.catalog]: This master's current role is: LEARNER
I20260810 04:07:03.337033 6744 catalog_manager.cc:1259] Loaded cluster ID: 1dd3f94c8dd942a4acce7f7524739ff2
I20260810 04:07:03.337081 6744 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2138c4c9465e4cd19eeef783497f22e5: loading cluster ID for follower catalog manager: success
I20260810 04:07:03.338234 6744 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 2138c4c9465e4cd19eeef783497f22e5: acquiring CA information for follower catalog manager: success
I20260810 04:07:03.338522 6744 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 2138c4c9465e4cd19eeef783497f22e5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 04:07:03.346607 6745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:03.338004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48574 (local address 127.31.32.61:40815)
0810 04:07:03.340234 (+ 2230us) server_negotiation.cc:206] Beginning negotiation
0810 04:07:03.340237 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:07:03.340245 (+ 8us) server_negotiation.cc:407] Connection header received
0810 04:07:03.340265 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:07:03.340266 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:07:03.340274 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:07:03.340361 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:07:03.340934 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:03.341556 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:03.342276 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:03.342485 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:03.342558 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:03.343797 (+ 1239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:07:03.343799 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:07:03.343799 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:07:03.343807 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:07:03.345482 (+ 1675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:07:03.345821 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:07:03.345823 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:07:03.345823 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:07:03.345891 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:07:03.346084 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:07:03.346085 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:07:03.346086 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:07:03.346218 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:07:03.346331 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0810 04:07:03.346415 (+ 84us) server_negotiation.cc:299] Negotiation successful
0810 04:07:03.346498 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2212,"thread_start_us":44,"threads_started":1}
WARNING: no policy specified for oryx/127.31.32.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.32.60@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.32.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/krb5kdc/oryx_127.31.32.60.keytab.
Entry for principal oryx/127.31.32.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/krb5kdc/oryx_127.31.32.60.keytab.
I20260810 04:07:03.379209 31872 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskws89Io/build/release/bin/kudu
/tmp/dist-test-taskws89Io/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.32.60:33905
--webserver_interface=127.31.32.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/krb5kdc/oryx_127.31.32.60.keytab
--principal=oryx/127.31.32.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.32.20:35351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.32.62:35131,127.31.32.61:40815,127.31.32.60:33905
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:07:03.474926 6748 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:07:03.475116 6748 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:07:03.475139 6748 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:07:03.476400 6748 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:07:03.476440 6748 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:07:03.476454 6748 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:07:03.476466 6748 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:07:03.476478 6748 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:07:03.478066 6748 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.32.20:35351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.31.32.62:35131,127.31.32.61:40815,127.31.32.60:33905
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/krb5kdc/oryx_127.31.32.60.keytab
--principal=oryx/127.31.32.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.32.60:33905
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.31.32.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6748
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:07:03.478366 6748 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:07:03.478600 6748 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 04:07:03.483485 6748 server_base.cc:1034] running on GCE node
W20260810 04:07:03.485324 6755 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:07:03.485318 6758 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:07:03.485456 6756 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:07:03.485738 6748 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:07:03.486032 6748 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:07:03.487205 6748 hybrid_clock.cc:648] HybridClock initialized: now 1786334823487184 us; error 37 us; skew 500 ppm
Aug 10 04:07:03 dist-test-slave-8w9v krb5kdc[5977](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334823, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.32.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:07:03.489773 6748 init.cc:377] Logged in from keytab as oryx/127.31.32.60@KRBTEST.COM (short username oryx)
I20260810 04:07:03.490254 6748 webserver.cc:460] Webserver started at http://127.31.32.60:42467/ using document root <none> and password file <none>
I20260810 04:07:03.490521 6748 fs_manager.cc:359] Metadata directory not provided
I20260810 04:07:03.490592 6748 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:07:03.490741 6748 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:07:03.491776 6748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/minicluster-data/master-2/data/instance:
uuid: "8b612a0b090e4e71a02370b642ea30ab"
format_stamp: "Formatted at 2026-08-10 04:07:03 on dist-test-slave-8w9v"
server_key: "e0ecc226a82c675510e5a24d1932b1a1"
server_key_iv: "fd0f4635a85b22d23cd8d5648a7be2ea"
server_key_version: "encryptionkey@0"
I20260810 04:07:03.492200 6748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/minicluster-data/master-2/wal/instance:
uuid: "8b612a0b090e4e71a02370b642ea30ab"
format_stamp: "Formatted at 2026-08-10 04:07:03 on dist-test-slave-8w9v"
server_key: "e0ecc226a82c675510e5a24d1932b1a1"
server_key_iv: "fd0f4635a85b22d23cd8d5648a7be2ea"
server_key_version: "encryptionkey@0"
I20260810 04:07:03.493570 6748 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:07:03.494259 6765 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:07:03.494422 6748 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 04:07:03.494529 6748 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/minicluster-data/master-2/data,/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/minicluster-data/master-2/wal
uuid: "8b612a0b090e4e71a02370b642ea30ab"
format_stamp: "Formatted at 2026-08-10 04:07:03 on dist-test-slave-8w9v"
server_key: "e0ecc226a82c675510e5a24d1932b1a1"
server_key_iv: "fd0f4635a85b22d23cd8d5648a7be2ea"
server_key_version: "encryptionkey@0"
I20260810 04:07:03.494612 6748 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:07:03.522256 6748 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:07:03.524097 6748 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:07:03.524304 6748 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:07:03.528594 6748 rpc_server.cc:307] RPC server started. Bound to: 127.31.32.60:33905
I20260810 04:07:03.528649 6819 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.32.60:33905 every 8 connection(s)
I20260810 04:07:03.529078 6748 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/minicluster-data/master-2/data/info.pb
I20260810 04:07:03.530133 6820 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:07:03.531721 6820 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.32.62" port: 35131 } has no permanent_uuid. Determining permanent_uuid...
I20260810 04:07:03.535579 31872 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskws89Io/build/release/bin/kudu as pid 6748
Aug 10 04:07:03 dist-test-slave-8w9v krb5kdc[5977](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334823, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.32.60@KRBTEST.COM for oryx/127.31.32.62@KRBTEST.COM
Aug 10 04:07:03 dist-test-slave-8w9v krb5kdc[5977](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334822, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.31.32.60@KRBTEST.COM
I20260810 04:07:03.542663 6078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:03.532578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50150 (local address 127.31.32.62:35131)
0810 04:07:03.532663 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0810 04:07:03.532666 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:07:03.532830 (+ 164us) server_negotiation.cc:407] Connection header received
0810 04:07:03.533343 (+ 513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:07:03.533344 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:07:03.533357 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:07:03.533406 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:07:03.534022 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:03.534531 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:03.535602 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:03.535840 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:03.535952 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:03.539123 (+ 3171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:07:03.539124 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:07:03.539124 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:07:03.539132 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:07:03.541167 (+ 2035us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:07:03.541586 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:07:03.541587 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:07:03.541588 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:07:03.541656 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:07:03.541979 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:07:03.541980 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:07:03.541981 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:07:03.542155 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:07:03.542267 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0810 04:07:03.542521 (+ 254us) server_negotiation.cc:299] Negotiation successful
0810 04:07:03.542594 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 04:07:03.542889 6822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:03.532476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:35131 (local address 127.0.0.1:50150)
0810 04:07:03.532706 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0810 04:07:03.532747 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 04:07:03.533229 (+ 482us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:07:03.533473 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:07:03.533484 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:07:03.533724 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:07:03.533909 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:03.533913 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:03.534637 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:03.534638 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:03.535519 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:03.535521 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:03.536031 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:03.536033 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:03.536266 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:03.536905 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:07:03.536916 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:07:03.539026 (+ 2110us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:07:03.541258 (+ 2232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:07:03.541265 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:07:03.541267 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:07:03.541503 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:07:03.541737 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:07:03.541737 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:07:03.541738 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:07:03.541888 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:07:03.542242 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:07:03.542246 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:07:03.542353 (+ 107us) client_negotiation.cc:769] Sending connection context
0810 04:07:03.542468 (+ 115us) client_negotiation.cc:240] Negotiation successful
0810 04:07:03.542629 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":124,"threads_started":1}
I20260810 04:07:03.543462 6820 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.32.61" port: 40815 } has no permanent_uuid. Determining permanent_uuid...
I20260810 04:07:03.546129 6823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:03.536284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53312 (local address 127.31.32.60:33905)
0810 04:07:03.536480 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0810 04:07:03.536485 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 04:07:03.536509 (+ 24us) server_negotiation.cc:407] Connection header received
0810 04:07:03.536531 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:07:03.536533 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:07:03.536557 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:07:03.536649 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:07:03.537704 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:03.538514 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:03.539434 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:03.539774 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:03.539838 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:03.542435 (+ 2597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:07:03.542441 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:07:03.542441 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:07:03.542448 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:07:03.544711 (+ 2263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:07:03.545154 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:07:03.545157 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:07:03.545157 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:07:03.545231 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:07:03.545477 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:07:03.545478 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:07:03.545479 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:07:03.545640 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:07:03.545817 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0810 04:07:03.545914 (+ 97us) server_negotiation.cc:299] Negotiation successful
0810 04:07:03.545999 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":119,"threads_started":1}
Aug 10 04:07:03 dist-test-slave-8w9v krb5kdc[5977](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334823, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.32.60@KRBTEST.COM for oryx/127.31.32.61@KRBTEST.COM
I20260810 04:07:03.550557 6822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:03.543722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.61:40815 (local address 127.0.0.1:33976)
0810 04:07:03.543884 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0810 04:07:03.543896 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 04:07:03.543957 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:07:03.544113 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:07:03.544114 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:07:03.544295 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:07:03.544359 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:03.544362 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:03.545024 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:03.545025 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:03.545491 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:03.545493 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:03.545877 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:03.545877 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:03.545951 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:03.546350 (+ 399us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:07:03.546355 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:07:03.547775 (+ 1420us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:07:03.549684 (+ 1909us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:07:03.549685 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:07:03.549686 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:07:03.549833 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:07:03.550002 (+ 169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:07:03.550003 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:07:03.550003 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:07:03.550050 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:07:03.550344 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:07:03.550344 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:07:03.550378 (+ 34us) client_negotiation.cc:769] Sending connection context
0810 04:07:03.550429 (+ 51us) client_negotiation.cc:240] Negotiation successful
0810 04:07:03.550501 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141}
I20260810 04:07:03.550637 6745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:03.543798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33976 (local address 127.31.32.61:40815)
0810 04:07:03.543878 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0810 04:07:03.543880 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:07:03.543940 (+ 60us) server_negotiation.cc:407] Connection header received
0810 04:07:03.544036 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:07:03.544037 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:07:03.544050 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:07:03.544103 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:07:03.544446 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:03.544902 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:03.545576 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:03.545794 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:03.545876 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:03.547865 (+ 1989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:07:03.547867 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:07:03.547867 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:07:03.547875 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:07:03.549611 (+ 1736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:07:03.549884 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:07:03.549884 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:07:03.549885 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:07:03.549936 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:07:03.550090 (+ 154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:07:03.550090 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:07:03.550091 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:07:03.550259 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:07:03.550337 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0810 04:07:03.550454 (+ 117us) server_negotiation.cc:299] Negotiation successful
0810 04:07:03.550532 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260810 04:07:03.551059 6820 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.32.60" port: 33905 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 04:07:03 dist-test-slave-8w9v krb5kdc[5977](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334823, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.32.60@KRBTEST.COM for oryx/127.31.32.60@KRBTEST.COM
I20260810 04:07:03.557410 6822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:03.551335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.60:33905 (local address 127.0.0.1:53318)
0810 04:07:03.551364 (+ 29us) negotiation.cc:107] Waiting for socket to connect
0810 04:07:03.551374 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 04:07:03.551425 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:07:03.551601 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:07:03.551602 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:07:03.551706 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:07:03.551732 (+ 26us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:03.551733 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:03.552494 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:03.552495 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:03.552927 (+ 432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:03.552928 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:03.553306 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:03.553307 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:03.553341 (+ 34us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:03.553663 (+ 322us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:07:03.553667 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:07:03.554765 (+ 1098us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:07:03.556670 (+ 1905us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:07:03.556671 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:07:03.556671 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:07:03.556816 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:07:03.556974 (+ 158us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:07:03.556975 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:07:03.556975 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:07:03.557022 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:07:03.557208 (+ 186us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:07:03.557209 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:07:03.557238 (+ 29us) client_negotiation.cc:769] Sending connection context
0810 04:07:03.557288 (+ 50us) client_negotiation.cc:240] Negotiation successful
0810 04:07:03.557354 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15}
I20260810 04:07:03.557497 6823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:03.551435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53318 (local address 127.31.32.60:33905)
0810 04:07:03.551491 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0810 04:07:03.551493 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:07:03.551502 (+ 9us) server_negotiation.cc:407] Connection header received
0810 04:07:03.551520 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:07:03.551520 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:07:03.551531 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:07:03.551591 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:07:03.551808 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:03.552419 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:03.552999 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:03.553222 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:03.553300 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:03.554851 (+ 1551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:07:03.554852 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:07:03.554852 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:07:03.554859 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:07:03.556600 (+ 1741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:07:03.556871 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:07:03.556872 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:07:03.556872 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:07:03.556921 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:07:03.557063 (+ 142us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:07:03.557063 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:07:03.557063 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:07:03.557156 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:07:03.557209 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0810 04:07:03.557306 (+ 97us) server_negotiation.cc:299] Negotiation successful
0810 04:07:03.557405 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":36,"server-negotiator.queue_time_us":36}
I20260810 04:07:03.559089 6820 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b612a0b090e4e71a02370b642ea30ab: Bootstrap starting.
I20260810 04:07:03.560034 6820 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8b612a0b090e4e71a02370b642ea30ab: Neither blocks nor log segments found. Creating new log.
I20260810 04:07:03.560490 6820 log.cc:824] T 00000000000000000000000000000000 P 8b612a0b090e4e71a02370b642ea30ab: Log is configured to *not* fsync() on all Append() calls
I20260810 04:07:03.561260 6820 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b612a0b090e4e71a02370b642ea30ab: No bootstrap required, opened a new log
I20260810 04:07:03.563005 6820 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8b612a0b090e4e71a02370b642ea30ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b858c2bd3c2346bf9ac073d925ee5ef1" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 35131 } } peers { permanent_uuid: "2138c4c9465e4cd19eeef783497f22e5" member_type: VOTER last_known_addr { host: "127.31.32.61" port: 40815 } } peers { permanent_uuid: "8b612a0b090e4e71a02370b642ea30ab" member_type: VOTER last_known_addr { host: "127.31.32.60" port: 33905 } }
I20260810 04:07:03.563140 6820 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8b612a0b090e4e71a02370b642ea30ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:07:03.563167 6820 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8b612a0b090e4e71a02370b642ea30ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b612a0b090e4e71a02370b642ea30ab, State: Initialized, Role: FOLLOWER
I20260810 04:07:03.563270 6820 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8b612a0b090e4e71a02370b642ea30ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b858c2bd3c2346bf9ac073d925ee5ef1" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 35131 } } peers { permanent_uuid: "2138c4c9465e4cd19eeef783497f22e5" member_type: VOTER last_known_addr { host: "127.31.32.61" port: 40815 } } peers { permanent_uuid: "8b612a0b090e4e71a02370b642ea30ab" member_type: VOTER last_known_addr { host: "127.31.32.60" port: 33905 } }
I20260810 04:07:03.563609 6820 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8b612a0b090e4e71a02370b642ea30ab [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:07:03.563560 6825 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b612a0b090e4e71a02370b642ea30ab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b858c2bd3c2346bf9ac073d925ee5ef1" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 35131 } } peers { permanent_uuid: "2138c4c9465e4cd19eeef783497f22e5" member_type: VOTER last_known_addr { host: "127.31.32.61" port: 40815 } } peers { permanent_uuid: "8b612a0b090e4e71a02370b642ea30ab" member_type: VOTER last_known_addr { host: "127.31.32.60" port: 33905 } } }
I20260810 04:07:03.563689 6825 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b612a0b090e4e71a02370b642ea30ab [sys.catalog]: This master's current role is: FOLLOWER
W20260810 04:07:03.565894 6836 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8b612a0b090e4e71a02370b642ea30ab: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 04:07:03.565955 6836 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 04:07:03.568602 6748 master_runner.cc:428] Detected that this master 8b612a0b090e4e71a02370b642ea30ab is joining an existing cluster
I20260810 04:07:03.568672 6748 master_runner.cc:432] Initiating AddMaster RPC to add 127.31.32.60:33905
I20260810 04:07:03.599498 31872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskws89Io/build/release/bin/kudu with pid 6100
I20260810 04:07:03.600842 6745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:03.570089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33986 (local address 127.31.32.61:40815)
0810 04:07:03.570129 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0810 04:07:03.570131 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:07:03.570478 (+ 347us) server_negotiation.cc:407] Connection header received
0810 04:07:03.570510 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:07:03.570511 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:07:03.570524 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:07:03.570605 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:07:03.571036 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:03.571822 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:03.573148 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:03.573440 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:03.573495 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:03.597197 (+ 23702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:07:03.597199 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:07:03.597199 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:07:03.597208 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:07:03.599229 (+ 2021us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:07:03.599716 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:07:03.599717 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:07:03.599718 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:07:03.599790 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:07:03.599988 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:07:03.599989 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:07:03.599989 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:07:03.600217 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:07:03.600281 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0810 04:07:03.600567 (+ 286us) server_negotiation.cc:299] Negotiation successful
0810 04:07:03.600676 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260810 04:07:03.601210 6078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:03.569868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50162 (local address 127.31.32.62:35131)
0810 04:07:03.569938 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0810 04:07:03.569941 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:07:03.570142 (+ 201us) server_negotiation.cc:407] Connection header received
0810 04:07:03.570290 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:07:03.570292 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:07:03.570307 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:07:03.570350 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:07:03.571065 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:03.571946 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:03.573185 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:03.573536 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:03.573585 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:03.597684 (+ 24099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:07:03.597686 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:07:03.597686 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:07:03.597695 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:07:03.599951 (+ 2256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:07:03.600402 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:07:03.600403 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:07:03.600404 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:07:03.600458 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:07:03.600665 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:07:03.600666 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:07:03.600666 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:07:03.600795 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:07:03.600845 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0810 04:07:03.601028 (+ 183us) server_negotiation.cc:299] Negotiation successful
0810 04:07:03.601112 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260810 04:07:03.601536 6844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:03.569798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:35131 (local address 127.0.0.1:50162)
0810 04:07:03.570069 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0810 04:07:03.570081 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 04:07:03.570174 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:07:03.570463 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:07:03.570464 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:07:03.570758 (+ 294us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:07:03.570950 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:03.570952 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:03.572057 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:03.572058 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:03.573063 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:03.573065 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:03.573680 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:03.573681 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:03.573748 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:03.596778 (+ 23030us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:07:03.596784 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:07:03.597568 (+ 784us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:07:03.600065 (+ 2497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:07:03.600066 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:07:03.600067 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:07:03.600315 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:07:03.600542 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:07:03.600543 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:07:03.600543 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:07:03.600600 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:07:03.600877 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:07:03.600878 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:07:03.600921 (+ 43us) client_negotiation.cc:769] Sending connection context
0810 04:07:03.601295 (+ 374us) client_negotiation.cc:240] Negotiation successful
0810 04:07:03.601416 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":144,"threads_started":1}
I20260810 04:07:03.602048 6845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:03.569888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.61:40815 (local address 127.0.0.1:33986)
0810 04:07:03.570378 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0810 04:07:03.570388 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 04:07:03.570452 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:07:03.570609 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:07:03.570610 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:07:03.570872 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:07:03.570935 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:03.570937 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:03.571932 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:03.571933 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:03.573023 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:03.573043 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:03.573589 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:03.573590 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:03.573650 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:03.574084 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:07:03.574089 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:07:03.597061 (+ 22972us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:07:03.599321 (+ 2260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:07:03.599322 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:07:03.599323 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:07:03.599524 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:07:03.599864 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:07:03.599865 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:07:03.599865 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:07:03.599925 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:07:03.600343 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:07:03.600344 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:07:03.600422 (+ 78us) client_negotiation.cc:769] Sending connection context
0810 04:07:03.601489 (+ 1067us) client_negotiation.cc:240] Negotiation successful
0810 04:07:03.601872 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"mutex_wait_us":42,"thread_start_us":118,"threads_started":1}
I20260810 04:07:03.605665 6823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:03.570146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53322 (local address 127.31.32.60:33905)
0810 04:07:03.570188 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0810 04:07:03.570190 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:07:03.570762 (+ 572us) server_negotiation.cc:407] Connection header received
0810 04:07:03.570890 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:07:03.570891 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:07:03.570901 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:07:03.570949 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:07:03.571389 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:03.572177 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:03.573281 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:03.573598 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:03.573642 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:03.597189 (+ 23547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:07:03.597191 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:07:03.597192 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:07:03.597199 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:07:03.599348 (+ 2149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:07:03.604980 (+ 5632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:07:03.604981 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:07:03.604982 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:07:03.605028 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:07:03.605191 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:07:03.605192 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:07:03.605192 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:07:03.605298 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:07:03.605334 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0810 04:07:03.605506 (+ 172us) server_negotiation.cc:299] Negotiation successful
0810 04:07:03.605578 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260810 04:07:03.606106 6843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:03.570205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.60:33905 (local address 127.0.0.1:53322)
0810 04:07:03.570683 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0810 04:07:03.570695 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 04:07:03.570787 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:07:03.571010 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:07:03.571011 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:07:03.571226 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:07:03.571286 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:03.571288 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:03.572273 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:03.572274 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:03.573186 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:03.573188 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:03.573714 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:03.573715 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:03.574086 (+ 371us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:03.574563 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:07:03.574569 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:07:03.597075 (+ 22506us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:07:03.604685 (+ 7610us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:07:03.604686 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:07:03.604687 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:07:03.604896 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:07:03.605089 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:07:03.605089 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:07:03.605090 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:07:03.605134 (+ 44us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:07:03.605370 (+ 236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:07:03.605370 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:07:03.605402 (+ 32us) client_negotiation.cc:769] Sending connection context
0810 04:07:03.605914 (+ 512us) client_negotiation.cc:240] Negotiation successful
0810 04:07:03.606007 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":93,"threads_started":1}
I20260810 04:07:03.612511 31872 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskws89Io/build/release/bin/kudu
/tmp/dist-test-taskws89Io/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.32.1:46215
--local_ip_for_outbound_sockets=127.31.32.1
--tserver_master_addrs=127.31.32.62:35131
--webserver_port=37967
--webserver_interface=127.31.32.1
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/krb5kdc/oryx_127.31.32.1.keytab
--principal=oryx/127.31.32.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.32.20:35351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.31.32.62:35131,127.31.32.61:40815
--tserver_master_addrs=127.31.32.62:35131,127.31.32.61:40815,127.31.32.60:33905 with env {KRB5CCNAME=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 04:07:03 dist-test-slave-8w9v krb5kdc[5977](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334822, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.32.62@KRBTEST.COM for oryx/127.31.32.60@KRBTEST.COM
I20260810 04:07:03.619336 6646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:03.609059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.60:33905 (local address 127.0.0.1:53330)
0810 04:07:03.609109 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0810 04:07:03.609120 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 04:07:03.609156 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:07:03.612096 (+ 2940us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:07:03.612097 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:07:03.612397 (+ 300us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:07:03.612455 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:03.612458 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:03.613367 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:03.613368 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:03.614236 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:03.614239 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:03.614610 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:03.614611 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:03.614691 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:03.615115 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:07:03.615120 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:07:03.616440 (+ 1320us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:07:03.618207 (+ 1767us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:07:03.618208 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:07:03.618208 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:07:03.618390 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:07:03.618596 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:07:03.618597 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:07:03.618598 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:07:03.618664 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:07:03.618998 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:07:03.619000 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:07:03.619044 (+ 44us) client_negotiation.cc:769] Sending connection context
0810 04:07:03.619125 (+ 81us) client_negotiation.cc:240] Negotiation successful
0810 04:07:03.619234 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260810 04:07:03.619336 6823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:03.609092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53330 (local address 127.31.32.60:33905)
0810 04:07:03.611961 (+ 2869us) server_negotiation.cc:206] Beginning negotiation
0810 04:07:03.611964 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:07:03.611972 (+ 8us) server_negotiation.cc:407] Connection header received
0810 04:07:03.612008 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:07:03.612009 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:07:03.612020 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:07:03.612080 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:07:03.612544 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:03.613273 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:03.614330 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:03.614520 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:03.614609 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:03.616530 (+ 1921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:07:03.616532 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:07:03.616533 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:07:03.616540 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:07:03.618127 (+ 1587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:07:03.618465 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:07:03.618466 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:07:03.618466 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:07:03.618520 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:07:03.618781 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:07:03.618782 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:07:03.618782 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:07:03.618915 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:07:03.618981 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0810 04:07:03.619166 (+ 185us) server_negotiation.cc:299] Negotiation successful
0810 04:07:03.619234 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2846}
I20260810 04:07:03.620424 6028 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.31.32.60:33905: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "8b612a0b090e4e71a02370b642ea30ab"
member_type: NON_VOTER
last_known_addr {
host: "127.31.32.60"
port: 33905
}
attrs {
promote: true
}
}
dest_uuid: "b858c2bd3c2346bf9ac073d925ee5ef1"
cas_config_opid_index: 5
I20260810 04:07:03.620719 6028 consensus_queue.cc:237] T 00000000000000000000000000000000 P b858c2bd3c2346bf9ac073d925ee5ef1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b858c2bd3c2346bf9ac073d925ee5ef1" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 35131 } } peers { permanent_uuid: "2138c4c9465e4cd19eeef783497f22e5" member_type: NON_VOTER last_known_addr { host: "127.31.32.61" port: 40815 } attrs { promote: true } } peers { permanent_uuid: "8b612a0b090e4e71a02370b642ea30ab" member_type: NON_VOTER last_known_addr { host: "127.31.32.60" port: 33905 } attrs { promote: true } }
I20260810 04:07:03.621096 6649 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b858c2bd3c2346bf9ac073d925ee5ef1 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 8b612a0b090e4e71a02370b642ea30ab (127.31.32.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b858c2bd3c2346bf9ac073d925ee5ef1" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 35131 } } peers { permanent_uuid: "2138c4c9465e4cd19eeef783497f22e5" member_type: NON_VOTER last_known_addr { host: "127.31.32.61" port: 40815 } attrs { promote: true } } peers { permanent_uuid: "8b612a0b090e4e71a02370b642ea30ab" member_type: NON_VOTER last_known_addr { host: "127.31.32.60" port: 33905 } attrs { promote: true } } }
I20260810 04:07:03.621980 6649 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.31.32.60:33905
I20260810 04:07:03.622057 6649 sys_catalog.cc:455] T 00000000000000000000000000000000 P b858c2bd3c2346bf9ac073d925ee5ef1 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "b858c2bd3c2346bf9ac073d925ee5ef1" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b858c2bd3c2346bf9ac073d925ee5ef1" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 35131 } } peers { permanent_uuid: "2138c4c9465e4cd19eeef783497f22e5" member_type: NON_VOTER last_known_addr { host: "127.31.32.61" port: 40815 } attrs { promote: true } } peers { permanent_uuid: "8b612a0b090e4e71a02370b642ea30ab" member_type: NON_VOTER last_known_addr { host: "127.31.32.60" port: 33905 } attrs { promote: true } } }
I20260810 04:07:03.622130 6649 sys_catalog.cc:458] T 00000000000000000000000000000000 P b858c2bd3c2346bf9ac073d925ee5ef1 [sys.catalog]: This master's current role is: LEADER
I20260810 04:07:03.623082 6793 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8b612a0b090e4e71a02370b642ea30ab [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:07:03.624179 6748 master.cc:537] Master@127.31.32.60:33905 shutting down...
I20260810 04:07:03.625154 6793 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8b612a0b090e4e71a02370b642ea30ab [term 1 FOLLOWER]: Refusing update from remote peer b858c2bd3c2346bf9ac073d925ee5ef1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260810 04:07:03.625847 6647 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b858c2bd3c2346bf9ac073d925ee5ef1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b612a0b090e4e71a02370b642ea30ab" member_type: NON_VOTER last_known_addr { host: "127.31.32.60" port: 33905 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260810 04:07:03.626837 6014 consensus_peers.cc:487] T 00000000000000000000000000000000 P b858c2bd3c2346bf9ac073d925ee5ef1 -> Peer 8b612a0b090e4e71a02370b642ea30ab (127.31.32.60:33905): Couldn't send request to peer 8b612a0b090e4e71a02370b642ea30ab. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260810 04:07:03.627759 6748 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 8b612a0b090e4e71a02370b642ea30ab [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 04:07:03.627851 6748 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 8b612a0b090e4e71a02370b642ea30ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:07:03.627893 6748 tablet_replica.cc:330] T 00000000000000000000000000000000 P 8b612a0b090e4e71a02370b642ea30ab: stopping tablet replica
I20260810 04:07:03.628695 6646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:03.622316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.61:40815 (local address 127.0.0.1:33988)
0810 04:07:03.622663 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0810 04:07:03.622675 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 04:07:03.622721 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:07:03.622987 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:07:03.622989 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:07:03.623034 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:07:03.623079 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:03.623081 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:03.624039 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:03.624040 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:03.627513 (+ 3473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:03.627517 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:03.628436 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:03.628437 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:03.628505 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:03.628528 (+ 23us) client_negotiation.cc:769] Sending connection context
0810 04:07:03.628586 (+ 58us) client_negotiation.cc:240] Negotiation successful
0810 04:07:03.628630 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329}
I20260810 04:07:03.628979 6745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:03.622387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33988 (local address 127.31.32.61:40815)
0810 04:07:03.622431 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0810 04:07:03.622433 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:07:03.622788 (+ 355us) server_negotiation.cc:407] Connection header received
0810 04:07:03.622824 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:07:03.622825 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:07:03.622851 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:07:03.622911 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:07:03.623158 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:03.623931 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:03.627615 (+ 3684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:03.628250 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:03.628321 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:03.628399 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0810 04:07:03.628851 (+ 452us) server_negotiation.cc:299] Negotiation successful
0810 04:07:03.628914 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260810 04:07:03.629187 6701 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2138c4c9465e4cd19eeef783497f22e5 [term 1 LEARNER]: Refusing update from remote peer b858c2bd3c2346bf9ac073d925ee5ef1: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260810 04:07:03.629393 6647 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b858c2bd3c2346bf9ac073d925ee5ef1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2138c4c9465e4cd19eeef783497f22e5" member_type: NON_VOTER last_known_addr { host: "127.31.32.61" port: 40815 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260810 04:07:03.630043 6701 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2138c4c9465e4cd19eeef783497f22e5 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 8b612a0b090e4e71a02370b642ea30ab (127.31.32.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b858c2bd3c2346bf9ac073d925ee5ef1" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 35131 } } peers { permanent_uuid: "2138c4c9465e4cd19eeef783497f22e5" member_type: NON_VOTER last_known_addr { host: "127.31.32.61" port: 40815 } attrs { promote: true } } peers { permanent_uuid: "8b612a0b090e4e71a02370b642ea30ab" member_type: NON_VOTER last_known_addr { host: "127.31.32.60" port: 33905 } attrs { promote: true } } }
I20260810 04:07:03.631421 6732 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2138c4c9465e4cd19eeef783497f22e5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b858c2bd3c2346bf9ac073d925ee5ef1. Latest consensus state: current_term: 1 leader_uuid: "b858c2bd3c2346bf9ac073d925ee5ef1" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b858c2bd3c2346bf9ac073d925ee5ef1" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 35131 } } peers { permanent_uuid: "2138c4c9465e4cd19eeef783497f22e5" member_type: NON_VOTER last_known_addr { host: "127.31.32.61" port: 40815 } attrs { promote: true } } peers { permanent_uuid: "8b612a0b090e4e71a02370b642ea30ab" member_type: NON_VOTER last_known_addr { host: "127.31.32.60" port: 33905 } attrs { promote: true } } }
I20260810 04:07:03.631533 6732 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2138c4c9465e4cd19eeef783497f22e5 [sys.catalog]: This master's current role is: LEARNER
I20260810 04:07:03.631572 6647 raft_consensus.cc:1050] T 00000000000000000000000000000000 P b858c2bd3c2346bf9ac073d925ee5ef1: attempting to promote NON_VOTER 2138c4c9465e4cd19eeef783497f22e5 to VOTER
I20260810 04:07:03.631575 6732 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2138c4c9465e4cd19eeef783497f22e5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b858c2bd3c2346bf9ac073d925ee5ef1" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b858c2bd3c2346bf9ac073d925ee5ef1" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 35131 } } peers { permanent_uuid: "2138c4c9465e4cd19eeef783497f22e5" member_type: NON_VOTER last_known_addr { host: "127.31.32.61" port: 40815 } attrs { promote: true } } peers { permanent_uuid: "8b612a0b090e4e71a02370b642ea30ab" member_type: NON_VOTER last_known_addr { host: "127.31.32.60" port: 33905 } attrs { promote: true } } }
I20260810 04:07:03.631618 6732 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2138c4c9465e4cd19eeef783497f22e5 [sys.catalog]: This master's current role is: LEARNER
I20260810 04:07:03.631749 6647 consensus_queue.cc:237] T 00000000000000000000000000000000 P b858c2bd3c2346bf9ac073d925ee5ef1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b858c2bd3c2346bf9ac073d925ee5ef1" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 35131 } } peers { permanent_uuid: "2138c4c9465e4cd19eeef783497f22e5" member_type: VOTER last_known_addr { host: "127.31.32.61" port: 40815 } attrs { promote: false } } peers { permanent_uuid: "8b612a0b090e4e71a02370b642ea30ab" member_type: NON_VOTER last_known_addr { host: "127.31.32.60" port: 33905 } attrs { promote: true } }
I20260810 04:07:03.632426 6701 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2138c4c9465e4cd19eeef783497f22e5 [term 1 LEARNER]: Refusing update from remote peer b858c2bd3c2346bf9ac073d925ee5ef1: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260810 04:07:03.632576 6646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:03.632238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.60:33905 (local address 127.0.0.1:53338)
0810 04:07:03.632447 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0810 04:07:03.632542 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.60:33905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":73,"threads_started":1}
I20260810 04:07:03.632617 6649 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b858c2bd3c2346bf9ac073d925ee5ef1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2138c4c9465e4cd19eeef783497f22e5" member_type: VOTER last_known_addr { host: "127.31.32.61" port: 40815 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260810 04:07:03.632978 6646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:03.632886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.60:33905 (local address 127.0.0.1:53340)
0810 04:07:03.632921 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0810 04:07:03.632942 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.60:33905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260810 04:07:03.633107 6014 consensus_peers.cc:487] T 00000000000000000000000000000000 P b858c2bd3c2346bf9ac073d925ee5ef1 -> Peer 8b612a0b090e4e71a02370b642ea30ab (127.31.32.60:33905): Couldn't send request to peer 8b612a0b090e4e71a02370b642ea30ab. Status: Network error: Client connection negotiation failed: client connection to 127.31.32.60:33905: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260810 04:07:03.633435 6650 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b858c2bd3c2346bf9ac073d925ee5ef1 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 2138c4c9465e4cd19eeef783497f22e5 (127.31.32.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b858c2bd3c2346bf9ac073d925ee5ef1" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 35131 } } peers { permanent_uuid: "2138c4c9465e4cd19eeef783497f22e5" member_type: VOTER last_known_addr { host: "127.31.32.61" port: 40815 } attrs { promote: false } } peers { permanent_uuid: "8b612a0b090e4e71a02370b642ea30ab" member_type: NON_VOTER last_known_addr { host: "127.31.32.60" port: 33905 } attrs { promote: true } } }
I20260810 04:07:03.634214 6650 sys_catalog.cc:455] T 00000000000000000000000000000000 P b858c2bd3c2346bf9ac073d925ee5ef1 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "b858c2bd3c2346bf9ac073d925ee5ef1" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b858c2bd3c2346bf9ac073d925ee5ef1" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 35131 } } peers { permanent_uuid: "2138c4c9465e4cd19eeef783497f22e5" member_type: VOTER last_known_addr { host: "127.31.32.61" port: 40815 } attrs { promote: false } } peers { permanent_uuid: "8b612a0b090e4e71a02370b642ea30ab" member_type: NON_VOTER last_known_addr { host: "127.31.32.60" port: 33905 } attrs { promote: true } } }
I20260810 04:07:03.634295 6650 sys_catalog.cc:458] T 00000000000000000000000000000000 P b858c2bd3c2346bf9ac073d925ee5ef1 [sys.catalog]: This master's current role is: LEADER
I20260810 04:07:03.634356 6650 sys_catalog.cc:455] T 00000000000000000000000000000000 P b858c2bd3c2346bf9ac073d925ee5ef1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b858c2bd3c2346bf9ac073d925ee5ef1" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b858c2bd3c2346bf9ac073d925ee5ef1" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 35131 } } peers { permanent_uuid: "2138c4c9465e4cd19eeef783497f22e5" member_type: VOTER last_known_addr { host: "127.31.32.61" port: 40815 } attrs { promote: false } } peers { permanent_uuid: "8b612a0b090e4e71a02370b642ea30ab" member_type: NON_VOTER last_known_addr { host: "127.31.32.60" port: 33905 } attrs { promote: true } } }
I20260810 04:07:03.634404 6650 sys_catalog.cc:458] T 00000000000000000000000000000000 P b858c2bd3c2346bf9ac073d925ee5ef1 [sys.catalog]: This master's current role is: LEADER
I20260810 04:07:03.634653 6701 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2138c4c9465e4cd19eeef783497f22e5 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 2138c4c9465e4cd19eeef783497f22e5 (127.31.32.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b858c2bd3c2346bf9ac073d925ee5ef1" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 35131 } } peers { permanent_uuid: "2138c4c9465e4cd19eeef783497f22e5" member_type: VOTER last_known_addr { host: "127.31.32.61" port: 40815 } attrs { promote: false } } peers { permanent_uuid: "8b612a0b090e4e71a02370b642ea30ab" member_type: NON_VOTER last_known_addr { host: "127.31.32.60" port: 33905 } attrs { promote: true } } }
I20260810 04:07:03.635452 6848 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2138c4c9465e4cd19eeef783497f22e5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b858c2bd3c2346bf9ac073d925ee5ef1" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b858c2bd3c2346bf9ac073d925ee5ef1" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 35131 } } peers { permanent_uuid: "2138c4c9465e4cd19eeef783497f22e5" member_type: VOTER last_known_addr { host: "127.31.32.61" port: 40815 } attrs { promote: false } } peers { permanent_uuid: "8b612a0b090e4e71a02370b642ea30ab" member_type: NON_VOTER last_known_addr { host: "127.31.32.60" port: 33905 } attrs { promote: true } } }
I20260810 04:07:03.635535 6848 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2138c4c9465e4cd19eeef783497f22e5 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 04:07:03.641311 6748 master.cc:552] Master@127.31.32.60:33905 shutdown complete.
I20260810 04:07:03.641381 6748 master_runner.cc:305] Clearing existing system tablet
I20260810 04:07:03.641513 6748 fs_manager.cc:359] Metadata directory not provided
I20260810 04:07:03.641575 6748 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:07:03.642194 6748 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.001s
I20260810 04:07:03.642694 6851 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:07:03.642829 6748 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:07:03.642876 6748 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260810 04:07:03.642904 6748 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/minicluster-data/master-2/data,/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/minicluster-data/master-2/wal
uuid: "8b612a0b090e4e71a02370b642ea30ab"
format_stamp: "Formatted at 2026-08-10 04:07:03 on dist-test-slave-8w9v"
server_key: "e0ecc226a82c675510e5a24d1932b1a1"
server_key_iv: "fd0f4635a85b22d23cd8d5648a7be2ea"
server_key_version: "encryptionkey@0"
I20260810 04:07:03.643497 6748 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 8b612a0b090e4e71a02370b642ea30ab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:07:03.644733 6748 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 8b612a0b090e4e71a02370b642ea30ab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260810 04:07:03.644816 6748 log.cc:1196] T 00000000000000000000000000000000 P 8b612a0b090e4e71a02370b642ea30ab: Deleting WAL directory at /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260810 04:07:03.645041 6748 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 8b612a0b090e4e71a02370b642ea30ab: Deleting consensus metadata
I20260810 04:07:03.645210 6748 master_runner.cc:315] Copying system tablet from 127.31.32.62:35131
I20260810 04:07:03.645886 6748 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 8b612a0b090e4e71a02370b642ea30ab: tablet copy: Beginning tablet copy session from remote peer at address 127.31.32.62:35131
I20260810 04:07:03.652344 6857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:03.646207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:35131 (local address 127.0.0.1:50168)
0810 04:07:03.646319 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0810 04:07:03.646328 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 04:07:03.646421 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:07:03.646550 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:07:03.646551 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:07:03.646725 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:07:03.646771 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:03.646773 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:03.647482 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:03.647483 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:03.648028 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:03.648030 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:03.648642 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:03.648643 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:03.648694 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:03.649193 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:07:03.649198 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:07:03.649492 (+ 294us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:07:03.651288 (+ 1796us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:07:03.651289 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:07:03.651289 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:07:03.651474 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:07:03.651680 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:07:03.651681 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:07:03.651682 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:07:03.651746 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:07:03.651990 (+ 244us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:07:03.651991 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:07:03.652035 (+ 44us) client_negotiation.cc:769] Sending connection context
0810 04:07:03.652118 (+ 83us) client_negotiation.cc:240] Negotiation successful
0810 04:07:03.652250 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":63,"threads_started":1}
I20260810 04:07:03.652344 6078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:03.646233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50168 (local address 127.31.32.62:35131)
0810 04:07:03.646277 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0810 04:07:03.646279 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:07:03.646407 (+ 128us) server_negotiation.cc:407] Connection header received
0810 04:07:03.646480 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:07:03.646481 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:07:03.646491 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:07:03.646546 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:07:03.646855 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:03.647410 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:03.648115 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:03.648423 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:03.648543 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:03.649601 (+ 1058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:07:03.649603 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:07:03.649603 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:07:03.649610 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:07:03.651191 (+ 1581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:07:03.651549 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:07:03.651551 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:07:03.651551 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:07:03.651615 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:07:03.651814 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:07:03.651815 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:07:03.651815 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:07:03.651927 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:07:03.651998 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0810 04:07:03.652181 (+ 183us) server_negotiation.cc:299] Negotiation successful
0810 04:07:03.652250 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260810 04:07:03.652958 6051 tablet_copy_service.cc:140] P b858c2bd3c2346bf9ac073d925ee5ef1: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 8b612a0b090e4e71a02370b642ea30ab ({username='oryx', principal='oryx/127.31.32.60@KRBTEST.COM'} at 127.0.0.1:50168)
I20260810 04:07:03.653036 6051 tablet_copy_service.cc:161] P b858c2bd3c2346bf9ac073d925ee5ef1: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 8b612a0b090e4e71a02370b642ea30ab at {username='oryx', principal='oryx/127.31.32.60@KRBTEST.COM'} at 127.0.0.1:50168: session id = 8b612a0b090e4e71a02370b642ea30ab-00000000000000000000000000000000
I20260810 04:07:03.653475 6051 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P b858c2bd3c2346bf9ac073d925ee5ef1: Tablet Copy: opened 0 blocks and 1 log segments
I20260810 04:07:03.653915 6748 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260810 04:07:03.654882 6748 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 8b612a0b090e4e71a02370b642ea30ab: tablet copy: Starting download of 0 data blocks...
I20260810 04:07:03.655033 6748 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 8b612a0b090e4e71a02370b642ea30ab: tablet copy: Starting download of 1 WAL segments...
I20260810 04:07:03.655684 6748 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 8b612a0b090e4e71a02370b642ea30ab: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260810 04:07:03.656450 6051 tablet_copy_service.cc:342] P b858c2bd3c2346bf9ac073d925ee5ef1: Request end of tablet copy session 8b612a0b090e4e71a02370b642ea30ab-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.31.32.60@KRBTEST.COM'} at 127.0.0.1:50168
I20260810 04:07:03.656503 6051 tablet_copy_service.cc:433] P b858c2bd3c2346bf9ac073d925ee5ef1: ending tablet copy session 8b612a0b090e4e71a02370b642ea30ab-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 8b612a0b090e4e71a02370b642ea30ab
I20260810 04:07:03.658720 6748 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:07:03.660017 6862 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:07:03.660027 6748 server_base.cc:1034] running on GCE node
W20260810 04:07:03.664222 6858 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:07:03.664749 6859 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:07:03.664920 6748 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:07:03.665089 6748 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:07:03.666239 6748 hybrid_clock.cc:648] HybridClock initialized: now 1786334823666198 us; error 56 us; skew 500 ppm
Aug 10 04:07:03 dist-test-slave-8w9v krb5kdc[5977](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334823, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.32.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:07:03.667640 6748 init.cc:377] Logged in from keytab as oryx/127.31.32.60@KRBTEST.COM (short username oryx)
I20260810 04:07:03.667918 6748 webserver.cc:460] Webserver started at http://127.31.32.60:46377/ using document root <none> and password file <none>
I20260810 04:07:03.668040 6748 fs_manager.cc:359] Metadata directory not provided
I20260810 04:07:03.668078 6748 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:07:03.668864 6748 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260810 04:07:03.669371 6869 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:07:03.669483 6748 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260810 04:07:03.669543 6748 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/minicluster-data/master-2/data,/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/minicluster-data/master-2/wal
uuid: "8b612a0b090e4e71a02370b642ea30ab"
format_stamp: "Formatted at 2026-08-10 04:07:03 on dist-test-slave-8w9v"
server_key: "e0ecc226a82c675510e5a24d1932b1a1"
server_key_iv: "fd0f4635a85b22d23cd8d5648a7be2ea"
server_key_version: "encryptionkey@0"
I20260810 04:07:03.669601 6748 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:07:03.682192 6748 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:07:03.683861 6748 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:07:03.686950 6748 rpc_server.cc:307] RPC server started. Bound to: 127.31.32.60:33905
I20260810 04:07:03.687256 6931 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.32.60:33905 every 8 connection(s)
I20260810 04:07:03.687358 6748 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/minicluster-data/master-2/data/info.pb
I20260810 04:07:03.688184 6932 sys_catalog.cc:263] Verifying existing consensus state
I20260810 04:07:03.688573 6932 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b612a0b090e4e71a02370b642ea30ab: Bootstrap starting.
I20260810 04:07:03.690726 6932 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b612a0b090e4e71a02370b642ea30ab: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 04:07:03.690974 6932 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b612a0b090e4e71a02370b642ea30ab: Bootstrap complete.
I20260810 04:07:03.691180 6932 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8b612a0b090e4e71a02370b642ea30ab [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b858c2bd3c2346bf9ac073d925ee5ef1" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 35131 } } peers { permanent_uuid: "2138c4c9465e4cd19eeef783497f22e5" member_type: VOTER last_known_addr { host: "127.31.32.61" port: 40815 } attrs { promote: false } } peers { permanent_uuid: "8b612a0b090e4e71a02370b642ea30ab" member_type: NON_VOTER last_known_addr { host: "127.31.32.60" port: 33905 } attrs { promote: true } }
I20260810 04:07:03.691267 6932 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8b612a0b090e4e71a02370b642ea30ab [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8b612a0b090e4e71a02370b642ea30ab, State: Initialized, Role: LEARNER
I20260810 04:07:03.691314 6932 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8b612a0b090e4e71a02370b642ea30ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b858c2bd3c2346bf9ac073d925ee5ef1" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 35131 } } peers { permanent_uuid: "2138c4c9465e4cd19eeef783497f22e5" member_type: VOTER last_known_addr { host: "127.31.32.61" port: 40815 } attrs { promote: false } } peers { permanent_uuid: "8b612a0b090e4e71a02370b642ea30ab" member_type: NON_VOTER last_known_addr { host: "127.31.32.60" port: 33905 } attrs { promote: true } }
I20260810 04:07:03.691524 6932 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8b612a0b090e4e71a02370b642ea30ab [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:07:03.691485 6935 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b612a0b090e4e71a02370b642ea30ab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b858c2bd3c2346bf9ac073d925ee5ef1" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 35131 } } peers { permanent_uuid: "2138c4c9465e4cd19eeef783497f22e5" member_type: VOTER last_known_addr { host: "127.31.32.61" port: 40815 } attrs { promote: false } } peers { permanent_uuid: "8b612a0b090e4e71a02370b642ea30ab" member_type: NON_VOTER last_known_addr { host: "127.31.32.60" port: 33905 } attrs { promote: true } } }
I20260810 04:07:03.691731 6935 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b612a0b090e4e71a02370b642ea30ab [sys.catalog]: This master's current role is: LEARNER
I20260810 04:07:03.692974 6946 catalog_manager.cc:1259] Loaded cluster ID: 1dd3f94c8dd942a4acce7f7524739ff2
I20260810 04:07:03.693064 6946 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8b612a0b090e4e71a02370b642ea30ab: loading cluster ID for follower catalog manager: success
I20260810 04:07:03.694064 6946 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8b612a0b090e4e71a02370b642ea30ab: acquiring CA information for follower catalog manager: success
I20260810 04:07:03.694363 6946 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8b612a0b090e4e71a02370b642ea30ab: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260810 04:07:03.735421 6846 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:07:03.735704 6846 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:07:03.735741 6846 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:07:03.737540 6846 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:07:03.737602 6846 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:07:03.737653 6846 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.32.1
I20260810 04:07:03.739883 6846 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.32.20:35351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/krb5kdc/oryx_127.31.32.1.keytab
--principal=oryx/127.31.32.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.32.1:46215
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.32.1
--webserver_port=37967
--tserver_master_addrs=127.31.32.62:35131,127.31.32.61:40815,127.31.32.60:33905
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6846
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.32.1
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:07:03.740233 6846 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:07:03.740545 6846 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:07:03.746677 6951 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:07:03.746894 6846 server_base.cc:1034] running on GCE node
W20260810 04:07:03.747082 6954 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:07:03.746953 6952 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:07:03.747368 6846 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:07:03.747629 6846 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:07:03.748785 6846 hybrid_clock.cc:648] HybridClock initialized: now 1786334823748764 us; error 49 us; skew 500 ppm
Aug 10 04:07:03 dist-test-slave-8w9v krb5kdc[5977](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334823, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.32.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:07:03.752240 6846 init.cc:377] Logged in from keytab as oryx/127.31.32.1@KRBTEST.COM (short username oryx)
I20260810 04:07:03.752669 6846 webserver.cc:460] Webserver started at http://127.31.32.1:37967/ using document root <none> and password file <none>
I20260810 04:07:03.752928 6846 fs_manager.cc:359] Metadata directory not provided
I20260810 04:07:03.753014 6846 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:07:03.754487 6846 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:07:03.755118 6963 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:07:03.755352 6846 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:07:03.755453 6846 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/minicluster-data/ts-0/data,/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/minicluster-data/ts-0/wal
uuid: "ec8b4c39ed3f419d99b5387ca3e51ca6"
format_stamp: "Formatted at 2026-08-10 04:07:02 on dist-test-slave-8w9v"
server_key: "76da7e23259e206e4fc803fd8467b6ea"
server_key_iv: "3a0efacd3c371a71b556b76e3d72ca7d"
server_key_version: "encryptionkey@0"
I20260810 04:07:03.755807 6846 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:07:03.767866 6846 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:07:03.770231 6846 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:07:03.770430 6846 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:07:03.770641 6846 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:07:03.770937 6846 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:07:03.770970 6846 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:07:03.771023 6846 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:07:03.771049 6846 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:07:03.779448 6846 rpc_server.cc:307] RPC server started. Bound to: 127.31.32.1:46215
I20260810 04:07:03.779496 7087 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.32.1:46215 every 8 connection(s)
I20260810 04:07:03.779843 6846 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/minicluster-data/ts-0/data/info.pb
Aug 10 04:07:03 dist-test-slave-8w9v krb5kdc[5977](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334823, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.32.1@KRBTEST.COM for oryx/127.31.32.61@KRBTEST.COM
Aug 10 04:07:03 dist-test-slave-8w9v krb5kdc[5977](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334823, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.32.1@KRBTEST.COM for oryx/127.31.32.62@KRBTEST.COM
Aug 10 04:07:03 dist-test-slave-8w9v krb5kdc[5977](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334823, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.32.1@KRBTEST.COM for oryx/127.31.32.60@KRBTEST.COM
I20260810 04:07:03.790032 31872 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskws89Io/build/release/bin/kudu as pid 6846
I20260810 04:07:03.790212 31872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskws89Io/build/release/bin/kudu with pid 6245
I20260810 04:07:03.795308 7096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:03.780673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.60:33905 (local address 127.31.32.1:54489)
0810 04:07:03.781175 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0810 04:07:03.781185 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 04:07:03.782214 (+ 1029us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:07:03.782521 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:07:03.782521 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:07:03.782965 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:07:03.783166 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:03.783172 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:03.784102 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:03.784103 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:03.785390 (+ 1287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:03.785392 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:03.785844 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:03.785845 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:03.785905 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:03.786579 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:07:03.786584 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:07:03.789212 (+ 2628us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:07:03.793552 (+ 4340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:07:03.793559 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:07:03.793561 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:07:03.793849 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:07:03.794077 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:07:03.794078 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:07:03.794078 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:07:03.794209 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:07:03.794550 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:07:03.794564 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:07:03.794744 (+ 180us) client_negotiation.cc:769] Sending connection context
0810 04:07:03.794856 (+ 112us) client_negotiation.cc:240] Negotiation successful
0810 04:07:03.795041 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"mutex_wait_us":113,"thread_start_us":98,"threads_started":1}
I20260810 04:07:03.795759 7095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:03.780820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.1:54489 (local address 127.31.32.60:33905)
0810 04:07:03.782396 (+ 1576us) server_negotiation.cc:206] Beginning negotiation
0810 04:07:03.782398 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:07:03.782408 (+ 10us) server_negotiation.cc:407] Connection header received
0810 04:07:03.782435 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:07:03.782436 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:07:03.782449 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:07:03.782506 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:07:03.783306 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:03.783984 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:03.785457 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:03.785771 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:03.785847 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:03.789321 (+ 3474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:07:03.789322 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:07:03.789323 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:07:03.789330 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:07:03.793466 (+ 4136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:07:03.793937 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:07:03.793938 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:07:03.793938 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:07:03.794004 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:07:03.794287 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:07:03.794289 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:07:03.794289 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:07:03.794456 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:07:03.794521 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0810 04:07:03.795567 (+ 1046us) server_negotiation.cc:299] Negotiation successful
0810 04:07:03.795661 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1554,"thread_start_us":57,"threads_started":1}
I20260810 04:07:03.795759 6078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:03.781097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.1:43343 (local address 127.31.32.62:35131)
0810 04:07:03.781142 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0810 04:07:03.781144 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:07:03.782451 (+ 1307us) server_negotiation.cc:407] Connection header received
0810 04:07:03.782555 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:07:03.782556 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:07:03.782567 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:07:03.782594 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:07:03.783303 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:03.784099 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:03.785406 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:03.785667 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:03.785749 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:03.789330 (+ 3581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:07:03.789332 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:07:03.789332 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:07:03.789340 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:07:03.794545 (+ 5205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:07:03.794940 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:07:03.794942 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:07:03.794942 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:07:03.795003 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:07:03.795199 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:07:03.795200 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:07:03.795200 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:07:03.795322 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:07:03.795370 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0810 04:07:03.795598 (+ 228us) server_negotiation.cc:299] Negotiation successful
0810 04:07:03.795674 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260810 04:07:03.796135 7093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:03.781002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:35131 (local address 127.31.32.1:43343)
0810 04:07:03.782391 (+ 1389us) negotiation.cc:107] Waiting for socket to connect
0810 04:07:03.782401 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 04:07:03.782477 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:07:03.782656 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:07:03.782657 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:07:03.782948 (+ 291us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:07:03.783165 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:03.783170 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:03.784194 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:03.784195 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:03.785292 (+ 1097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:03.785296 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:03.785750 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:03.785751 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:03.785830 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:03.786429 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:07:03.786434 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:07:03.789220 (+ 2786us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:07:03.794631 (+ 5411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:07:03.794631 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:07:03.794632 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:07:03.794857 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:07:03.795078 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:07:03.795079 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:07:03.795079 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:07:03.795132 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:07:03.795422 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:07:03.795423 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:07:03.795479 (+ 56us) client_negotiation.cc:769] Sending connection context
0810 04:07:03.795887 (+ 408us) client_negotiation.cc:240] Negotiation successful
0810 04:07:03.796011 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1375,"mutex_wait_us":107,"thread_start_us":67,"threads_started":1}
I20260810 04:07:03.796157 7088 heartbeater.cc:347] Connected to a master server at 127.31.32.60:33905
I20260810 04:07:03.796243 7088 heartbeater.cc:464] Registering TS with master...
I20260810 04:07:03.796444 7089 heartbeater.cc:347] Connected to a master server at 127.31.32.62:35131
I20260810 04:07:03.796489 7089 heartbeater.cc:464] Registering TS with master...
I20260810 04:07:03.796537 7089 heartbeater.cc:511] Master 127.31.32.62:35131 requested a full tablet report, sending...
I20260810 04:07:03.796453 7088 heartbeater.cc:511] Master 127.31.32.60:33905 requested a full tablet report, sending...
I20260810 04:07:03.796967 6028 ts_manager.cc:194] Re-registered known tserver with Master: ec8b4c39ed3f419d99b5387ca3e51ca6 (127.31.32.1:46215)
I20260810 04:07:03.797065 6890 ts_manager.cc:194] Registered new tserver with Master: ec8b4c39ed3f419d99b5387ca3e51ca6 (127.31.32.1:46215)
I20260810 04:07:03.797447 6028 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.31.32.1@KRBTEST.COM'} at 127.31.32.1:43343
I20260810 04:07:03.802027 6745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:03.780738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.1:59003 (local address 127.31.32.61:40815)
0810 04:07:03.780793 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0810 04:07:03.780795 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:07:03.781171 (+ 376us) server_negotiation.cc:407] Connection header received
0810 04:07:03.782310 (+ 1139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:07:03.782311 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:07:03.782337 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:07:03.782419 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:07:03.783323 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:03.784099 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:03.785194 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:03.785472 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:03.785540 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:03.788031 (+ 2491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:07:03.788033 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:07:03.788033 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:07:03.788040 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:07:03.789769 (+ 1729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:07:03.801042 (+ 11273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:07:03.801044 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:07:03.801044 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:07:03.801109 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:07:03.801335 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:07:03.801336 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:07:03.801337 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:07:03.801492 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:07:03.801564 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0810 04:07:03.801762 (+ 198us) server_negotiation.cc:299] Negotiation successful
0810 04:07:03.801872 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260810 04:07:03.802110 7094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:03.780673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.61:40815 (local address 127.31.32.1:59003)
0810 04:07:03.781012 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0810 04:07:03.781060 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0810 04:07:03.782211 (+ 1151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:07:03.782503 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:07:03.782507 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:07:03.782958 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:07:03.783166 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:03.783172 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:03.784222 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:03.784223 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:03.785097 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:03.785099 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:03.785556 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:03.785557 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:03.785703 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:03.786241 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:07:03.786251 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:07:03.787932 (+ 1681us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:07:03.800765 (+ 12833us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:07:03.800766 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:07:03.800766 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:07:03.800948 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:07:03.801179 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:07:03.801180 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:07:03.801180 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:07:03.801245 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:07:03.801577 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:07:03.801578 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:07:03.801621 (+ 43us) client_negotiation.cc:769] Sending connection context
0810 04:07:03.801701 (+ 80us) client_negotiation.cc:240] Negotiation successful
0810 04:07:03.801810 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"mutex_wait_us":96,"thread_start_us":102,"threads_started":1}
I20260810 04:07:03.803079 7090 heartbeater.cc:347] Connected to a master server at 127.31.32.61:40815
I20260810 04:07:03.803138 7090 heartbeater.cc:464] Registering TS with master...
I20260810 04:07:03.803229 7090 heartbeater.cc:511] Master 127.31.32.61:40815 requested a full tablet report, sending...
I20260810 04:07:03.803674 6691 ts_manager.cc:194] Registered new tserver with Master: ec8b4c39ed3f419d99b5387ca3e51ca6 (127.31.32.1:46215)
I20260810 04:07:03.807459 31872 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskws89Io/build/release/bin/kudu
/tmp/dist-test-taskws89Io/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.32.2:41349
--local_ip_for_outbound_sockets=127.31.32.2
--tserver_master_addrs=127.31.32.62:35131
--webserver_port=44995
--webserver_interface=127.31.32.2
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/krb5kdc/oryx_127.31.32.2.keytab
--principal=oryx/127.31.32.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.32.20:35351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.31.32.62:35131,127.31.32.61:40815
--tserver_master_addrs=127.31.32.62:35131,127.31.32.61:40815,127.31.32.60:33905 with env {KRB5CCNAME=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:07:03.891307 7097 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:07:03.891489 7097 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:07:03.891512 7097 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:07:03.892789 7097 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:07:03.892829 7097 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:07:03.892876 7097 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.32.2
I20260810 04:07:03.894387 7097 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.32.20:35351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/krb5kdc/oryx_127.31.32.2.keytab
--principal=oryx/127.31.32.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.32.2:41349
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.32.2
--webserver_port=44995
--tserver_master_addrs=127.31.32.62:35131,127.31.32.61:40815,127.31.32.60:33905
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7097
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.32.2
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:07:03.894577 7097 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:07:03.894763 7097 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:07:03.900028 7102 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:07:03.900110 7097 server_base.cc:1034] running on GCE node
W20260810 04:07:03.900076 7103 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:07:03.900055 7106 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:07:03.900440 7097 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:07:03.900725 7097 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:07:03.901867 7097 hybrid_clock.cc:648] HybridClock initialized: now 1786334823901848 us; error 36 us; skew 500 ppm
Aug 10 04:07:03 dist-test-slave-8w9v krb5kdc[5977](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334823, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.32.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:07:03.904415 7097 init.cc:377] Logged in from keytab as oryx/127.31.32.2@KRBTEST.COM (short username oryx)
I20260810 04:07:03.904850 7097 webserver.cc:460] Webserver started at http://127.31.32.2:44995/ using document root <none> and password file <none>
I20260810 04:07:03.905093 7097 fs_manager.cc:359] Metadata directory not provided
I20260810 04:07:03.905187 7097 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:07:03.906567 7097 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260810 04:07:03.907224 7113 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:07:03.907440 7097 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 04:07:03.907533 7097 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/minicluster-data/ts-1/data,/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/minicluster-data/ts-1/wal
uuid: "378cfcc378724919ac8f9f7d19b06bfa"
format_stamp: "Formatted at 2026-08-10 04:07:02 on dist-test-slave-8w9v"
server_key: "79c9be715fbc904f5ab9be749ad5b090"
server_key_iv: "71e4f8f11f6cbef90f36800ecad423d1"
server_key_version: "encryptionkey@0"
I20260810 04:07:03.907899 7097 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:07:03.923718 7097 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:07:03.925654 7097 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:07:03.925825 7097 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:07:03.926075 7097 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:07:03.926395 7097 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:07:03.926450 7097 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:07:03.926491 7097 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:07:03.926519 7097 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:07:03.932711 7097 rpc_server.cc:307] RPC server started. Bound to: 127.31.32.2:41349
I20260810 04:07:03.932749 7236 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.32.2:41349 every 8 connection(s)
I20260810 04:07:03.933137 7097 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/minicluster-data/ts-1/data/info.pb
Aug 10 04:07:03 dist-test-slave-8w9v krb5kdc[5977](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334823, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.32.2@KRBTEST.COM for oryx/127.31.32.62@KRBTEST.COM
Aug 10 04:07:03 dist-test-slave-8w9v krb5kdc[5977](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334823, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.32.2@KRBTEST.COM for oryx/127.31.32.60@KRBTEST.COM
Aug 10 04:07:03 dist-test-slave-8w9v krb5kdc[5977](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334823, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.32.2@KRBTEST.COM for oryx/127.31.32.61@KRBTEST.COM
I20260810 04:07:03.942744 31872 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskws89Io/build/release/bin/kudu as pid 7097
I20260810 04:07:03.942925 31872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskws89Io/build/release/bin/kudu with pid 6388
I20260810 04:07:03.944509 6078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:03.934408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.2:55741 (local address 127.31.32.62:35131)
0810 04:07:03.934455 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0810 04:07:03.934456 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 04:07:03.934593 (+ 137us) server_negotiation.cc:407] Connection header received
0810 04:07:03.935475 (+ 882us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:07:03.935476 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:07:03.935490 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:07:03.935542 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:07:03.936483 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:03.937225 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:03.938357 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:03.938657 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:03.938741 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:03.940982 (+ 2241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:07:03.940984 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:07:03.940984 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:07:03.940992 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:07:03.942833 (+ 1841us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:07:03.943364 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:07:03.943365 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:07:03.943366 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:07:03.943428 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:07:03.943714 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:07:03.943715 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:07:03.943715 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:07:03.943850 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:07:03.943897 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0810 04:07:03.944294 (+ 397us) server_negotiation.cc:299] Negotiation successful
0810 04:07:03.944387 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260810 04:07:03.945031 7241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:03.934210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:35131 (local address 127.31.32.2:55741)
0810 04:07:03.934515 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0810 04:07:03.934523 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0810 04:07:03.935299 (+ 776us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:07:03.935553 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:07:03.935554 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:07:03.936083 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:07:03.936342 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:03.936348 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:03.937325 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:03.937326 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:03.938281 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:03.938283 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:03.938741 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:03.938742 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:03.938827 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:03.939391 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:07:03.939395 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:07:03.940889 (+ 1494us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:07:03.942921 (+ 2032us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:07:03.942928 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:07:03.942930 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:07:03.943254 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:07:03.943506 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:07:03.943506 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:07:03.943507 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:07:03.943632 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:07:03.943946 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:07:03.943952 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:07:03.944106 (+ 154us) client_negotiation.cc:769] Sending connection context
0810 04:07:03.944614 (+ 508us) client_negotiation.cc:240] Negotiation successful
0810 04:07:03.944805 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"mutex_wait_us":32,"thread_start_us":65,"threads_started":1}
I20260810 04:07:03.945341 7244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:03.933842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.61:40815 (local address 127.31.32.2:47143)
0810 04:07:03.934426 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0810 04:07:03.934435 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 04:07:03.935299 (+ 864us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:07:03.935550 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:07:03.935554 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:07:03.936055 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:07:03.936342 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:03.936347 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:03.937332 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:03.937333 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:03.938338 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:03.938340 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:03.938898 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:03.938899 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:03.938962 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:03.939573 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:07:03.939578 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:07:03.941800 (+ 2222us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:07:03.944040 (+ 2240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:07:03.944040 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:07:03.944041 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:07:03.944268 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:07:03.944640 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:07:03.944640 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:07:03.944641 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:07:03.944714 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:07:03.945020 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:07:03.945020 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:07:03.945060 (+ 40us) client_negotiation.cc:769] Sending connection context
0810 04:07:03.945144 (+ 84us) client_negotiation.cc:240] Negotiation successful
0810 04:07:03.945243 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"mutex_wait_us":104,"thread_start_us":109,"threads_started":1}
I20260810 04:07:03.945361 6745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:03.934189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.2:47143 (local address 127.31.32.61:40815)
0810 04:07:03.934325 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0810 04:07:03.934327 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:07:03.934490 (+ 163us) server_negotiation.cc:407] Connection header received
0810 04:07:03.935387 (+ 897us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:07:03.935389 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:07:03.935403 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:07:03.935460 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:07:03.936473 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:03.937223 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:03.938458 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:03.938746 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:03.938839 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:03.941895 (+ 3056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:07:03.941897 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:07:03.941897 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:07:03.941905 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:07:03.943966 (+ 2061us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:07:03.944348 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:07:03.944349 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:07:03.944350 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:07:03.944407 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:07:03.944790 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:07:03.944791 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:07:03.944792 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:07:03.944934 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:07:03.944985 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0810 04:07:03.945188 (+ 203us) server_negotiation.cc:299] Negotiation successful
0810 04:07:03.945262 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20260810 04:07:03.946183 7239 heartbeater.cc:347] Connected to a master server at 127.31.32.61:40815
I20260810 04:07:03.946251 7239 heartbeater.cc:464] Registering TS with master...
I20260810 04:07:03.946398 7239 heartbeater.cc:511] Master 127.31.32.61:40815 requested a full tablet report, sending...
I20260810 04:07:03.946794 6691 ts_manager.cc:194] Registered new tserver with Master: 378cfcc378724919ac8f9f7d19b06bfa (127.31.32.2:41349)
I20260810 04:07:03.949102 7243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:03.933842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.60:33905 (local address 127.31.32.2:56947)
0810 04:07:03.934034 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0810 04:07:03.934069 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0810 04:07:03.935283 (+ 1214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:07:03.935553 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:07:03.935554 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:07:03.936083 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:07:03.936345 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:03.936348 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:03.937187 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:03.937188 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:03.938031 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:03.938033 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:03.938558 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:03.938559 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:03.938702 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:03.939220 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:07:03.939227 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:07:03.941744 (+ 2517us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:07:03.947848 (+ 6104us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:07:03.947849 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:07:03.947850 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:07:03.948142 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:07:03.948372 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:07:03.948372 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:07:03.948373 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:07:03.948441 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:07:03.948738 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:07:03.948739 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:07:03.948779 (+ 40us) client_negotiation.cc:769] Sending connection context
0810 04:07:03.948849 (+ 70us) client_negotiation.cc:240] Negotiation successful
0810 04:07:03.948942 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"mutex_wait_us":44,"thread_start_us":108,"threads_started":1}
I20260810 04:07:03.949164 7095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:03.934272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.2:56947 (local address 127.31.32.60:33905)
0810 04:07:03.934362 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0810 04:07:03.934365 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:07:03.934374 (+ 9us) server_negotiation.cc:407] Connection header received
0810 04:07:03.935439 (+ 1065us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:07:03.935441 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:07:03.935456 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:07:03.935487 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:07:03.936480 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:03.937026 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:03.938133 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:03.938478 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:03.938543 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:03.941844 (+ 3301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:07:03.941845 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:07:03.941846 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:07:03.941855 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:07:03.943606 (+ 1751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:07:03.948232 (+ 4626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:07:03.948233 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:07:03.948233 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:07:03.948297 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:07:03.948517 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:07:03.948518 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:07:03.948518 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:07:03.948673 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:07:03.948742 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0810 04:07:03.948927 (+ 185us) server_negotiation.cc:299] Negotiation successful
0810 04:07:03.949042 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260810 04:07:03.952633 7237 heartbeater.cc:347] Connected to a master server at 127.31.32.60:33905
I20260810 04:07:03.952641 7238 heartbeater.cc:347] Connected to a master server at 127.31.32.62:35131
I20260810 04:07:03.952696 7237 heartbeater.cc:464] Registering TS with master...
I20260810 04:07:03.952696 7238 heartbeater.cc:464] Registering TS with master...
I20260810 04:07:03.952780 7237 heartbeater.cc:511] Master 127.31.32.60:33905 requested a full tablet report, sending...
I20260810 04:07:03.952780 7238 heartbeater.cc:511] Master 127.31.32.62:35131 requested a full tablet report, sending...
I20260810 04:07:03.953027 6028 ts_manager.cc:194] Re-registered known tserver with Master: 378cfcc378724919ac8f9f7d19b06bfa (127.31.32.2:41349)
I20260810 04:07:03.953078 6890 ts_manager.cc:194] Registered new tserver with Master: 378cfcc378724919ac8f9f7d19b06bfa (127.31.32.2:41349)
I20260810 04:07:03.953416 6028 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.31.32.2@KRBTEST.COM'} at 127.31.32.2:55741
I20260810 04:07:03.957446 31872 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskws89Io/build/release/bin/kudu
/tmp/dist-test-taskws89Io/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.32.3:32929
--local_ip_for_outbound_sockets=127.31.32.3
--tserver_master_addrs=127.31.32.62:35131
--webserver_port=35853
--webserver_interface=127.31.32.3
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/krb5kdc/oryx_127.31.32.3.keytab
--principal=oryx/127.31.32.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.32.20:35351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.31.32.62:35131,127.31.32.61:40815
--tserver_master_addrs=127.31.32.62:35131,127.31.32.61:40815,127.31.32.60:33905 with env {KRB5CCNAME=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:07:04.044142 7245 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:07:04.044410 7245 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:07:04.044449 7245 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260810 04:07:04.044646 7095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:04.041527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53356 (local address 127.31.32.60:33905)
0810 04:07:04.041597 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0810 04:07:04.041599 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:07:04.041611 (+ 12us) server_negotiation.cc:407] Connection header received
0810 04:07:04.041645 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:07:04.041646 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:07:04.041677 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:07:04.041762 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:07:04.041954 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:04.042440 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:04.043849 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:04.044294 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:04.044351 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:04.044431 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0810 04:07:04.044547 (+ 116us) server_negotiation.cc:299] Negotiation successful
0810 04:07:04.044590 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260810 04:07:04.044652 6646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:04.041413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.60:33905 (local address 127.0.0.1:53356)
0810 04:07:04.041490 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0810 04:07:04.041503 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 04:07:04.041583 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:07:04.041806 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:07:04.041808 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:07:04.041810 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:07:04.041870 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:04.041873 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:04.042527 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:04.042528 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:04.043744 (+ 1216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:04.043748 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:04.044384 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:04.044386 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:04.044460 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:04.044465 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 04:07:04.044529 (+ 64us) client_negotiation.cc:240] Negotiation successful
0810 04:07:04.044579 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260810 04:07:04.045049 6905 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 8b612a0b090e4e71a02370b642ea30ab [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260810 04:07:04.045212 6935 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b612a0b090e4e71a02370b642ea30ab [sys.catalog]: SysCatalogTable state changed. Reason: New leader b858c2bd3c2346bf9ac073d925ee5ef1. Latest consensus state: current_term: 1 leader_uuid: "b858c2bd3c2346bf9ac073d925ee5ef1" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b858c2bd3c2346bf9ac073d925ee5ef1" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 35131 } } peers { permanent_uuid: "2138c4c9465e4cd19eeef783497f22e5" member_type: VOTER last_known_addr { host: "127.31.32.61" port: 40815 } attrs { promote: false } } peers { permanent_uuid: "8b612a0b090e4e71a02370b642ea30ab" member_type: NON_VOTER last_known_addr { host: "127.31.32.60" port: 33905 } attrs { promote: true } } }
I20260810 04:07:04.045328 6935 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b612a0b090e4e71a02370b642ea30ab [sys.catalog]: This master's current role is: LEARNER
W20260810 04:07:04.045991 7245 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:07:04.046034 7245 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:07:04.046062 7245 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.32.3
I20260810 04:07:04.047533 7245 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.32.20:35351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/krb5kdc/oryx_127.31.32.3.keytab
--principal=oryx/127.31.32.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.32.3:32929
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.32.3
--webserver_port=35853
--tserver_master_addrs=127.31.32.62:35131,127.31.32.61:40815,127.31.32.60:33905
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7245
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.32.3
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:07:04.047720 7245 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:07:04.047904 7245 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:07:04.052982 7252 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:07:04.052994 7254 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:07:04.053282 7245 server_base.cc:1034] running on GCE node
W20260810 04:07:04.052994 7251 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:07:04.053531 7245 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:07:04.053776 7245 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:07:04.054917 7245 hybrid_clock.cc:648] HybridClock initialized: now 1786334824054907 us; error 26 us; skew 500 ppm
Aug 10 04:07:04 dist-test-slave-8w9v krb5kdc[5977](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334824, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.32.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:07:04.057451 7245 init.cc:377] Logged in from keytab as oryx/127.31.32.3@KRBTEST.COM (short username oryx)
I20260810 04:07:04.057910 7245 webserver.cc:460] Webserver started at http://127.31.32.3:35853/ using document root <none> and password file <none>
I20260810 04:07:04.058141 7245 fs_manager.cc:359] Metadata directory not provided
I20260810 04:07:04.058228 7245 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:07:04.059636 7245 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:07:04.060364 7262 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:07:04.060544 7245 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:07:04.060652 7245 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/minicluster-data/ts-2/data,/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/minicluster-data/ts-2/wal
uuid: "71c25e47472544fa9e042cfcc7bf4f96"
format_stamp: "Formatted at 2026-08-10 04:07:03 on dist-test-slave-8w9v"
server_key: "6e5f63a4fc0713ed71eeebb6402a14ab"
server_key_iv: "509ed22b9999d7360da495e7fed07250"
server_key_version: "encryptionkey@0"
I20260810 04:07:04.061008 7245 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:07:04.072824 7245 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:07:04.074674 7245 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:07:04.074837 7245 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:07:04.075070 7245 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:07:04.075381 7245 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:07:04.075436 7245 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:07:04.075479 7245 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:07:04.075507 7245 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:07:04.081441 7245 rpc_server.cc:307] RPC server started. Bound to: 127.31.32.3:32929
I20260810 04:07:04.081492 7388 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.32.3:32929 every 8 connection(s)
I20260810 04:07:04.081854 7245 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/minicluster-data/ts-2/data/info.pb
I20260810 04:07:04.082942 31872 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskws89Io/build/release/bin/kudu as pid 7245
Aug 10 04:07:04 dist-test-slave-8w9v krb5kdc[5977](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334824, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.32.3@KRBTEST.COM for oryx/127.31.32.61@KRBTEST.COM
Aug 10 04:07:04 dist-test-slave-8w9v krb5kdc[5977](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334824, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.32.3@KRBTEST.COM for oryx/127.31.32.62@KRBTEST.COM
Aug 10 04:07:04 dist-test-slave-8w9v krb5kdc[5977](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334824, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.32.3@KRBTEST.COM for oryx/127.31.32.60@KRBTEST.COM
I20260810 04:07:04.095074 7400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:04.084410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34002 (local address 127.31.32.61:40815)
0810 04:07:04.084694 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0810 04:07:04.084696 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:07:04.084986 (+ 290us) server_negotiation.cc:407] Connection header received
0810 04:07:04.085104 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:07:04.085105 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:07:04.085121 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:07:04.085166 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:07:04.086238 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:04.087018 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:04.088067 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:04.088408 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:04.088457 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:04.091005 (+ 2548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:07:04.091006 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:07:04.091007 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:07:04.091014 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:07:04.093576 (+ 2562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:07:04.094040 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:07:04.094041 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:07:04.094042 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:07:04.094111 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:07:04.094380 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:07:04.094381 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:07:04.094382 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:07:04.094518 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:07:04.094599 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0810 04:07:04.094834 (+ 235us) server_negotiation.cc:299] Negotiation successful
0810 04:07:04.094920 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":61,"threads_started":1}
I20260810 04:07:04.095631 6078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:04.083008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.3:35065 (local address 127.31.32.62:35131)
0810 04:07:04.083055 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0810 04:07:04.083056 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 04:07:04.083066 (+ 10us) server_negotiation.cc:407] Connection header received
0810 04:07:04.084054 (+ 988us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:07:04.084057 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:07:04.084070 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:07:04.084140 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:07:04.086268 (+ 2128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:04.087013 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:04.087962 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:04.088295 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:04.088397 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:04.092283 (+ 3886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:07:04.092284 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:07:04.092284 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:07:04.092291 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:07:04.094044 (+ 1753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:07:04.094537 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:07:04.094538 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:07:04.094539 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:07:04.094599 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:07:04.094905 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:07:04.094906 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:07:04.094906 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:07:04.095030 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:07:04.095089 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0810 04:07:04.095431 (+ 342us) server_negotiation.cc:299] Negotiation successful
0810 04:07:04.095515 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260810 04:07:04.095846 7395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:04.082839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:35131 (local address 127.31.32.3:35065)
0810 04:07:04.082984 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0810 04:07:04.082994 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 04:07:04.083879 (+ 885us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:07:04.084236 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:07:04.084237 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:07:04.086106 (+ 1869us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:07:04.086182 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:04.086184 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:04.087104 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:04.087105 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:04.087878 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:04.087881 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:04.088399 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:04.088400 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:04.088471 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:04.089188 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:07:04.089200 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:07:04.092179 (+ 2979us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:07:04.094145 (+ 1966us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:07:04.094152 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:07:04.094154 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:07:04.094447 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:07:04.094674 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:07:04.094675 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:07:04.094675 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:07:04.094825 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:07:04.095117 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:07:04.095123 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:07:04.095292 (+ 169us) client_negotiation.cc:769] Sending connection context
0810 04:07:04.095400 (+ 108us) client_negotiation.cc:240] Negotiation successful
0810 04:07:04.095530 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":65,"threads_started":1}
I20260810 04:07:04.096334 7390 heartbeater.cc:347] Connected to a master server at 127.31.32.62:35131
I20260810 04:07:04.096421 7390 heartbeater.cc:464] Registering TS with master...
I20260810 04:07:04.096613 7390 heartbeater.cc:511] Master 127.31.32.62:35131 requested a full tablet report, sending...
I20260810 04:07:04.096834 7394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:04.082536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.61:40815 (local address 127.31.32.3:50323)
0810 04:07:04.082784 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0810 04:07:04.082795 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 04:07:04.083873 (+ 1078us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:07:04.084270 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:07:04.084272 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:07:04.085062 (+ 790us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:07:04.085121 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:04.085123 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:04.086035 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:04.086036 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:04.087391 (+ 1355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:04.087394 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:04.087931 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:04.087932 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:04.088042 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:04.089185 (+ 1143us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:07:04.089195 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:07:04.091215 (+ 2020us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:07:04.094634 (+ 3419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:07:04.094635 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:07:04.094635 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:07:04.094874 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:07:04.095093 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:07:04.095094 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:07:04.095094 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:07:04.095629 (+ 535us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:07:04.096372 (+ 743us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:07:04.096373 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:07:04.096420 (+ 47us) client_negotiation.cc:769] Sending connection context
0810 04:07:04.096488 (+ 68us) client_negotiation.cc:240] Negotiation successful
0810 04:07:04.096600 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"mutex_wait_us":56,"thread_start_us":99,"threads_started":1}
I20260810 04:07:04.096863 6745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:04.082581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.3:50323 (local address 127.31.32.61:40815)
0810 04:07:04.082689 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0810 04:07:04.082691 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:07:04.082876 (+ 185us) server_negotiation.cc:407] Connection header received
0810 04:07:04.084059 (+ 1183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:07:04.084061 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:07:04.084077 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:07:04.084157 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:07:04.085224 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:04.085962 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:04.087499 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:04.087809 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:04.087871 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:04.091754 (+ 3883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:07:04.091756 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:07:04.091756 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:07:04.091764 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:07:04.094549 (+ 2785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:07:04.094958 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:07:04.094959 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:07:04.094960 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:07:04.095019 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:07:04.095935 (+ 916us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:07:04.095936 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:07:04.095936 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:07:04.096241 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:07:04.096340 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0810 04:07:04.096666 (+ 326us) server_negotiation.cc:299] Negotiation successful
0810 04:07:04.096740 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260810 04:07:04.097093 6028 ts_manager.cc:194] Re-registered known tserver with Master: 71c25e47472544fa9e042cfcc7bf4f96 (127.31.32.3:32929)
I20260810 04:07:04.097271 7391 heartbeater.cc:347] Connected to a master server at 127.31.32.61:40815
I20260810 04:07:04.097321 7391 heartbeater.cc:464] Registering TS with master...
I20260810 04:07:04.097393 7391 heartbeater.cc:511] Master 127.31.32.61:40815 requested a full tablet report, sending...
I20260810 04:07:04.097640 6028 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.31.32.3@KRBTEST.COM'} at 127.31.32.3:35065
I20260810 04:07:04.097649 6691 ts_manager.cc:194] Registered new tserver with Master: 71c25e47472544fa9e042cfcc7bf4f96 (127.31.32.3:32929)
I20260810 04:07:04.099038 7393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:04.082535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.60:33905 (local address 127.31.32.3:57181)
0810 04:07:04.082702 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0810 04:07:04.082741 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0810 04:07:04.083875 (+ 1134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:07:04.084234 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:07:04.084237 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:07:04.084686 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:07:04.084967 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:04.084973 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:04.086016 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:04.086017 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:04.087386 (+ 1369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:04.087388 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:04.087896 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:04.087897 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:04.088039 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:04.089613 (+ 1574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:07:04.089896 (+ 283us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:07:04.093410 (+ 3514us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:07:04.095344 (+ 1934us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:07:04.095344 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:07:04.095345 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:07:04.098152 (+ 2807us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:07:04.098381 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:07:04.098382 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:07:04.098382 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:07:04.098441 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:07:04.098712 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:07:04.098713 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:07:04.098759 (+ 46us) client_negotiation.cc:769] Sending connection context
0810 04:07:04.098835 (+ 76us) client_negotiation.cc:240] Negotiation successful
0810 04:07:04.098942 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"mutex_wait_us":62,"thread_start_us":113,"threads_started":1}
I20260810 04:07:04.099110 7095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:04.082635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.3:57181 (local address 127.31.32.60:33905)
0810 04:07:04.082725 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0810 04:07:04.082727 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:07:04.082822 (+ 95us) server_negotiation.cc:407] Connection header received
0810 04:07:04.084013 (+ 1191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:07:04.084014 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:07:04.084033 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:07:04.084068 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:07:04.085079 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:04.085894 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:04.087478 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:04.087774 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:04.087814 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:04.093510 (+ 5696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:07:04.093511 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:07:04.093511 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:07:04.093518 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:07:04.095252 (+ 1734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:07:04.098227 (+ 2975us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:07:04.098229 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:07:04.098229 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:07:04.098299 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:07:04.098514 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:07:04.098515 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:07:04.098515 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:07:04.098633 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:07:04.098687 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0810 04:07:04.098920 (+ 233us) server_negotiation.cc:299] Negotiation successful
0810 04:07:04.099006 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260810 04:07:04.099447 7389 heartbeater.cc:347] Connected to a master server at 127.31.32.60:33905
I20260810 04:07:04.099510 7389 heartbeater.cc:464] Registering TS with master...
I20260810 04:07:04.099828 7389 heartbeater.cc:511] Master 127.31.32.60:33905 requested a full tablet report, sending...
I20260810 04:07:04.100210 6890 ts_manager.cc:194] Registered new tserver with Master: 71c25e47472544fa9e042cfcc7bf4f96 (127.31.32.3:32929)
I20260810 04:07:04.100849 7403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:04.084683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50176 (local address 127.31.32.62:35131)
0810 04:07:04.088478 (+ 3795us) server_negotiation.cc:206] Beginning negotiation
0810 04:07:04.088480 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:07:04.088486 (+ 6us) server_negotiation.cc:407] Connection header received
0810 04:07:04.088505 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:07:04.088506 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:07:04.088517 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:07:04.088621 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:07:04.089349 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:04.090075 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:04.091170 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:04.091492 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:04.091821 (+ 329us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:04.093039 (+ 1218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:07:04.093040 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:07:04.093040 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:07:04.093047 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:07:04.099475 (+ 6428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:07:04.099923 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:07:04.099925 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:07:04.099925 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:07:04.100007 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:07:04.100278 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:07:04.100279 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:07:04.100279 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:07:04.100429 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:07:04.100541 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0810 04:07:04.100645 (+ 104us) server_negotiation.cc:299] Negotiation successful
0810 04:07:04.100715 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3769,"thread_start_us":90,"threads_started":1}
I20260810 04:07:04.103402 7406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:04.084836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53364 (local address 127.31.32.60:33905)
0810 04:07:04.084953 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0810 04:07:04.084955 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:07:04.085056 (+ 101us) server_negotiation.cc:407] Connection header received
0810 04:07:04.085160 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:07:04.085161 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:07:04.085185 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:07:04.085227 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:07:04.088919 (+ 3692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:04.089690 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:04.098568 (+ 8878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:04.098863 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:04.098922 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:04.100746 (+ 1824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:07:04.100748 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:07:04.100748 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:07:04.100755 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:07:04.102221 (+ 1466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:07:04.102664 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:07:04.102665 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:07:04.102665 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:07:04.102711 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:07:04.102909 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:07:04.102910 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:07:04.102910 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:07:04.103011 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:07:04.103102 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0810 04:07:04.103222 (+ 120us) server_negotiation.cc:299] Negotiation successful
0810 04:07:04.103305 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94,"thread_start_us":63,"threads_started":1}
I20260810 04:07:04.102840 6028 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50176:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 04:07:04.104809 6028 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 04:07:04.105654 6648 sys_catalog.cc:455] T 00000000000000000000000000000000 P b858c2bd3c2346bf9ac073d925ee5ef1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b858c2bd3c2346bf9ac073d925ee5ef1" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b858c2bd3c2346bf9ac073d925ee5ef1" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 35131 } } peers { permanent_uuid: "2138c4c9465e4cd19eeef783497f22e5" member_type: VOTER last_known_addr { host: "127.31.32.61" port: 40815 } attrs { promote: false } } peers { permanent_uuid: "8b612a0b090e4e71a02370b642ea30ab" member_type: NON_VOTER last_known_addr { host: "127.31.32.60" port: 33905 } attrs { promote: true } } }
I20260810 04:07:04.105767 6648 sys_catalog.cc:458] T 00000000000000000000000000000000 P b858c2bd3c2346bf9ac073d925ee5ef1 [sys.catalog]: This master's current role is: LEADER
I20260810 04:07:04.107179 6648 raft_consensus.cc:1050] T 00000000000000000000000000000000 P b858c2bd3c2346bf9ac073d925ee5ef1: attempting to promote NON_VOTER 8b612a0b090e4e71a02370b642ea30ab to VOTER
I20260810 04:07:04.107331 6648 consensus_queue.cc:237] T 00000000000000000000000000000000 P b858c2bd3c2346bf9ac073d925ee5ef1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b858c2bd3c2346bf9ac073d925ee5ef1" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 35131 } } peers { permanent_uuid: "2138c4c9465e4cd19eeef783497f22e5" member_type: VOTER last_known_addr { host: "127.31.32.61" port: 40815 } attrs { promote: false } } peers { permanent_uuid: "8b612a0b090e4e71a02370b642ea30ab" member_type: VOTER last_known_addr { host: "127.31.32.60" port: 33905 } attrs { promote: false } }
I20260810 04:07:04.107707 6905 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8b612a0b090e4e71a02370b642ea30ab [term 1 LEARNER]: Refusing update from remote peer b858c2bd3c2346bf9ac073d925ee5ef1: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260810 04:07:04.107873 6701 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2138c4c9465e4cd19eeef783497f22e5 [term 1 FOLLOWER]: Refusing update from remote peer b858c2bd3c2346bf9ac073d925ee5ef1: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260810 04:07:04.108088 6647 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b858c2bd3c2346bf9ac073d925ee5ef1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b612a0b090e4e71a02370b642ea30ab" member_type: VOTER last_known_addr { host: "127.31.32.60" port: 33905 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260810 04:07:04.108280 6650 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b858c2bd3c2346bf9ac073d925ee5ef1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2138c4c9465e4cd19eeef783497f22e5" member_type: VOTER last_known_addr { host: "127.31.32.61" port: 40815 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260810 04:07:04.109230 6647 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b858c2bd3c2346bf9ac073d925ee5ef1 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 8b612a0b090e4e71a02370b642ea30ab (127.31.32.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b858c2bd3c2346bf9ac073d925ee5ef1" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 35131 } } peers { permanent_uuid: "2138c4c9465e4cd19eeef783497f22e5" member_type: VOTER last_known_addr { host: "127.31.32.61" port: 40815 } attrs { promote: false } } peers { permanent_uuid: "8b612a0b090e4e71a02370b642ea30ab" member_type: VOTER last_known_addr { host: "127.31.32.60" port: 33905 } attrs { promote: false } } }
I20260810 04:07:04.109431 6905 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 8b612a0b090e4e71a02370b642ea30ab [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 8b612a0b090e4e71a02370b642ea30ab (127.31.32.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b858c2bd3c2346bf9ac073d925ee5ef1" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 35131 } } peers { permanent_uuid: "2138c4c9465e4cd19eeef783497f22e5" member_type: VOTER last_known_addr { host: "127.31.32.61" port: 40815 } attrs { promote: false } } peers { permanent_uuid: "8b612a0b090e4e71a02370b642ea30ab" member_type: VOTER last_known_addr { host: "127.31.32.60" port: 33905 } attrs { promote: false } } }
I20260810 04:07:04.110226 6647 sys_catalog.cc:455] T 00000000000000000000000000000000 P b858c2bd3c2346bf9ac073d925ee5ef1 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "b858c2bd3c2346bf9ac073d925ee5ef1" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b858c2bd3c2346bf9ac073d925ee5ef1" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 35131 } } peers { permanent_uuid: "2138c4c9465e4cd19eeef783497f22e5" member_type: VOTER last_known_addr { host: "127.31.32.61" port: 40815 } attrs { promote: false } } peers { permanent_uuid: "8b612a0b090e4e71a02370b642ea30ab" member_type: VOTER last_known_addr { host: "127.31.32.60" port: 33905 } attrs { promote: false } } }
I20260810 04:07:04.110322 6647 sys_catalog.cc:458] T 00000000000000000000000000000000 P b858c2bd3c2346bf9ac073d925ee5ef1 [sys.catalog]: This master's current role is: LEADER
I20260810 04:07:04.110316 6935 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b612a0b090e4e71a02370b642ea30ab [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b858c2bd3c2346bf9ac073d925ee5ef1" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b858c2bd3c2346bf9ac073d925ee5ef1" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 35131 } } peers { permanent_uuid: "2138c4c9465e4cd19eeef783497f22e5" member_type: VOTER last_known_addr { host: "127.31.32.61" port: 40815 } attrs { promote: false } } peers { permanent_uuid: "8b612a0b090e4e71a02370b642ea30ab" member_type: VOTER last_known_addr { host: "127.31.32.60" port: 33905 } attrs { promote: false } } }
I20260810 04:07:04.110394 6935 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b612a0b090e4e71a02370b642ea30ab [sys.catalog]: This master's current role is: FOLLOWER
I20260810 04:07:04.110433 6650 sys_catalog.cc:455] T 00000000000000000000000000000000 P b858c2bd3c2346bf9ac073d925ee5ef1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b858c2bd3c2346bf9ac073d925ee5ef1" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b858c2bd3c2346bf9ac073d925ee5ef1" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 35131 } } peers { permanent_uuid: "2138c4c9465e4cd19eeef783497f22e5" member_type: VOTER last_known_addr { host: "127.31.32.61" port: 40815 } attrs { promote: false } } peers { permanent_uuid: "8b612a0b090e4e71a02370b642ea30ab" member_type: VOTER last_known_addr { host: "127.31.32.60" port: 33905 } attrs { promote: false } } }
I20260810 04:07:04.110529 6650 sys_catalog.cc:458] T 00000000000000000000000000000000 P b858c2bd3c2346bf9ac073d925ee5ef1 [sys.catalog]: This master's current role is: LEADER
I20260810 04:07:04.110810 6701 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2138c4c9465e4cd19eeef783497f22e5 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 8b612a0b090e4e71a02370b642ea30ab (127.31.32.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b858c2bd3c2346bf9ac073d925ee5ef1" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 35131 } } peers { permanent_uuid: "2138c4c9465e4cd19eeef783497f22e5" member_type: VOTER last_known_addr { host: "127.31.32.61" port: 40815 } attrs { promote: false } } peers { permanent_uuid: "8b612a0b090e4e71a02370b642ea30ab" member_type: VOTER last_known_addr { host: "127.31.32.60" port: 33905 } attrs { promote: false } } }
I20260810 04:07:04.111987 6848 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2138c4c9465e4cd19eeef783497f22e5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b858c2bd3c2346bf9ac073d925ee5ef1" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b858c2bd3c2346bf9ac073d925ee5ef1" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 35131 } } peers { permanent_uuid: "2138c4c9465e4cd19eeef783497f22e5" member_type: VOTER last_known_addr { host: "127.31.32.61" port: 40815 } attrs { promote: false } } peers { permanent_uuid: "8b612a0b090e4e71a02370b642ea30ab" member_type: VOTER last_known_addr { host: "127.31.32.60" port: 33905 } attrs { promote: false } } }
I20260810 04:07:04.112084 6848 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2138c4c9465e4cd19eeef783497f22e5 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 04:07:04.110450 6647 sys_catalog.cc:455] T 00000000000000000000000000000000 P b858c2bd3c2346bf9ac073d925ee5ef1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b858c2bd3c2346bf9ac073d925ee5ef1" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b858c2bd3c2346bf9ac073d925ee5ef1" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 35131 } } peers { permanent_uuid: "2138c4c9465e4cd19eeef783497f22e5" member_type: VOTER last_known_addr { host: "127.31.32.61" port: 40815 } attrs { promote: false } } peers { permanent_uuid: "8b612a0b090e4e71a02370b642ea30ab" member_type: VOTER last_known_addr { host: "127.31.32.60" port: 33905 } attrs { promote: false } } }
I20260810 04:07:04.112239 6647 sys_catalog.cc:458] T 00000000000000000000000000000000 P b858c2bd3c2346bf9ac073d925ee5ef1 [sys.catalog]: This master's current role is: LEADER
I20260810 04:07:04.115607 7417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:04.111757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.3:32929 (local address 127.0.0.1:38470)
0810 04:07:04.112346 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0810 04:07:04.112356 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 04:07:04.112414 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:07:04.112699 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:07:04.112700 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:07:04.112701 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:07:04.112750 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:04.112752 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:04.113688 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:04.113690 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:04.114512 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:04.114514 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:04.115389 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:04.115390 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:04.115452 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:04.115458 (+ 6us) client_negotiation.cc:769] Sending connection context
0810 04:07:04.115491 (+ 33us) client_negotiation.cc:240] Negotiation successful
0810 04:07:04.115528 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"thread_start_us":132,"threads_started":1}
I20260810 04:07:04.116313 6646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:04.111254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.2:41349 (local address 127.0.0.1:54562)
0810 04:07:04.111297 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0810 04:07:04.111308 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 04:07:04.111365 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:07:04.111789 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:07:04.111791 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:07:04.111792 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:07:04.111844 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:04.111846 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:04.113827 (+ 1981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:04.113829 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:04.114615 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:04.114618 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:04.116043 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:04.116044 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:04.116108 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:04.116112 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 04:07:04.116212 (+ 100us) client_negotiation.cc:240] Negotiation successful
0810 04:07:04.116252 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260810 04:07:04.116336 7418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:04.112014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38470 (local address 127.31.32.3:32929)
0810 04:07:04.112518 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0810 04:07:04.112522 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 04:07:04.112549 (+ 27us) server_negotiation.cc:407] Connection header received
0810 04:07:04.112579 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:07:04.112604 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:07:04.112621 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:07:04.112702 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:07:04.112839 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:04.113587 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:04.114605 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:04.115281 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:04.115789 (+ 508us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:04.115882 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0810 04:07:04.115911 (+ 29us) server_negotiation.cc:299] Negotiation successful
0810 04:07:04.115971 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":471,"thread_start_us":96,"threads_started":1}
I20260810 04:07:04.116351 7415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:04.111376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54562 (local address 127.31.32.2:41349)
0810 04:07:04.111595 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0810 04:07:04.111600 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 04:07:04.111611 (+ 11us) server_negotiation.cc:407] Connection header received
0810 04:07:04.111654 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:07:04.111656 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:07:04.111674 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:07:04.112484 (+ 810us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:07:04.112566 (+ 82us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:04.113731 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:04.115354 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:04.115929 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:04.116043 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:04.116137 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0810 04:07:04.116226 (+ 89us) server_negotiation.cc:299] Negotiation successful
0810 04:07:04.116265 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":101,"threads_started":1}
I20260810 04:07:04.116526 6849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:04.111351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.1:46215 (local address 127.0.0.1:35818)
0810 04:07:04.111509 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0810 04:07:04.111520 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 04:07:04.111575 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:07:04.111857 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:07:04.111858 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:07:04.111859 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:07:04.111976 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:04.111978 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:04.113585 (+ 1607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:04.113586 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:04.115234 (+ 1648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:04.115236 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:04.116270 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:04.116271 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:04.116360 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:04.116365 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 04:07:04.116401 (+ 36us) client_negotiation.cc:240] Negotiation successful
0810 04:07:04.116449 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"mutex_wait_us":770}
I20260810 04:07:04.116974 7416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:04.111502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35818 (local address 127.31.32.1:46215)
0810 04:07:04.111681 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0810 04:07:04.111685 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 04:07:04.111694 (+ 9us) server_negotiation.cc:407] Connection header received
0810 04:07:04.111732 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:07:04.111734 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:07:04.111750 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:07:04.112116 (+ 366us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:07:04.112214 (+ 98us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:04.113490 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:04.115345 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:04.116100 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:04.116711 (+ 611us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:04.116811 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0810 04:07:04.116841 (+ 30us) server_negotiation.cc:299] Negotiation successful
0810 04:07:04.116885 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":79,"threads_started":1}
I20260810 04:07:04.117050 7315 tablet_service.cc:1467] Processing CreateTablet for tablet 7e854480a8364c0195285b9fac067783 (DEFAULT_TABLE table=test-table [id=b54cd5a75470494fbe53ce9f332ceab6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:07:04.117401 7164 tablet_service.cc:1467] Processing CreateTablet for tablet 7e854480a8364c0195285b9fac067783 (DEFAULT_TABLE table=test-table [id=b54cd5a75470494fbe53ce9f332ceab6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:07:04.117498 7315 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e854480a8364c0195285b9fac067783. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:07:04.117578 7016 tablet_service.cc:1467] Processing CreateTablet for tablet 7e854480a8364c0195285b9fac067783 (DEFAULT_TABLE table=test-table [id=b54cd5a75470494fbe53ce9f332ceab6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:07:04.117753 7164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e854480a8364c0195285b9fac067783. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:07:04.117913 7016 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e854480a8364c0195285b9fac067783. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:07:04.120715 7419 tablet_bootstrap.cc:492] T 7e854480a8364c0195285b9fac067783 P 378cfcc378724919ac8f9f7d19b06bfa: Bootstrap starting.
I20260810 04:07:04.121254 7420 tablet_bootstrap.cc:492] T 7e854480a8364c0195285b9fac067783 P 71c25e47472544fa9e042cfcc7bf4f96: Bootstrap starting.
I20260810 04:07:04.121359 7419 tablet_bootstrap.cc:654] T 7e854480a8364c0195285b9fac067783 P 378cfcc378724919ac8f9f7d19b06bfa: Neither blocks nor log segments found. Creating new log.
I20260810 04:07:04.121652 7421 tablet_bootstrap.cc:492] T 7e854480a8364c0195285b9fac067783 P ec8b4c39ed3f419d99b5387ca3e51ca6: Bootstrap starting.
I20260810 04:07:04.121701 7419 log.cc:824] T 7e854480a8364c0195285b9fac067783 P 378cfcc378724919ac8f9f7d19b06bfa: Log is configured to *not* fsync() on all Append() calls
I20260810 04:07:04.122054 7420 tablet_bootstrap.cc:654] T 7e854480a8364c0195285b9fac067783 P 71c25e47472544fa9e042cfcc7bf4f96: Neither blocks nor log segments found. Creating new log.
I20260810 04:07:04.122359 7421 tablet_bootstrap.cc:654] T 7e854480a8364c0195285b9fac067783 P ec8b4c39ed3f419d99b5387ca3e51ca6: Neither blocks nor log segments found. Creating new log.
I20260810 04:07:04.122402 7420 log.cc:824] T 7e854480a8364c0195285b9fac067783 P 71c25e47472544fa9e042cfcc7bf4f96: Log is configured to *not* fsync() on all Append() calls
I20260810 04:07:04.122713 7419 tablet_bootstrap.cc:492] T 7e854480a8364c0195285b9fac067783 P 378cfcc378724919ac8f9f7d19b06bfa: No bootstrap required, opened a new log
I20260810 04:07:04.122709 7421 log.cc:824] T 7e854480a8364c0195285b9fac067783 P ec8b4c39ed3f419d99b5387ca3e51ca6: Log is configured to *not* fsync() on all Append() calls
I20260810 04:07:04.122798 7419 ts_tablet_manager.cc:1397] T 7e854480a8364c0195285b9fac067783 P 378cfcc378724919ac8f9f7d19b06bfa: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 04:07:04.123222 7420 tablet_bootstrap.cc:492] T 7e854480a8364c0195285b9fac067783 P 71c25e47472544fa9e042cfcc7bf4f96: No bootstrap required, opened a new log
I20260810 04:07:04.123292 7420 ts_tablet_manager.cc:1397] T 7e854480a8364c0195285b9fac067783 P 71c25e47472544fa9e042cfcc7bf4f96: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 04:07:04.123492 7421 tablet_bootstrap.cc:492] T 7e854480a8364c0195285b9fac067783 P ec8b4c39ed3f419d99b5387ca3e51ca6: No bootstrap required, opened a new log
I20260810 04:07:04.123574 7421 ts_tablet_manager.cc:1397] T 7e854480a8364c0195285b9fac067783 P ec8b4c39ed3f419d99b5387ca3e51ca6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 04:07:04.125140 7421 raft_consensus.cc:348] T 7e854480a8364c0195285b9fac067783 P ec8b4c39ed3f419d99b5387ca3e51ca6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "378cfcc378724919ac8f9f7d19b06bfa" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 41349 } } peers { permanent_uuid: "ec8b4c39ed3f419d99b5387ca3e51ca6" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 46215 } } peers { permanent_uuid: "71c25e47472544fa9e042cfcc7bf4f96" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 32929 } }
I20260810 04:07:04.125301 7421 raft_consensus.cc:374] T 7e854480a8364c0195285b9fac067783 P ec8b4c39ed3f419d99b5387ca3e51ca6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:07:04.125348 7421 raft_consensus.cc:729] T 7e854480a8364c0195285b9fac067783 P ec8b4c39ed3f419d99b5387ca3e51ca6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec8b4c39ed3f419d99b5387ca3e51ca6, State: Initialized, Role: FOLLOWER
I20260810 04:07:04.125478 7421 consensus_queue.cc:260] T 7e854480a8364c0195285b9fac067783 P ec8b4c39ed3f419d99b5387ca3e51ca6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "378cfcc378724919ac8f9f7d19b06bfa" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 41349 } } peers { permanent_uuid: "ec8b4c39ed3f419d99b5387ca3e51ca6" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 46215 } } peers { permanent_uuid: "71c25e47472544fa9e042cfcc7bf4f96" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 32929 } }
I20260810 04:07:04.125427 7420 raft_consensus.cc:348] T 7e854480a8364c0195285b9fac067783 P 71c25e47472544fa9e042cfcc7bf4f96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "378cfcc378724919ac8f9f7d19b06bfa" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 41349 } } peers { permanent_uuid: "ec8b4c39ed3f419d99b5387ca3e51ca6" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 46215 } } peers { permanent_uuid: "71c25e47472544fa9e042cfcc7bf4f96" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 32929 } }
I20260810 04:07:04.125460 7419 raft_consensus.cc:348] T 7e854480a8364c0195285b9fac067783 P 378cfcc378724919ac8f9f7d19b06bfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "378cfcc378724919ac8f9f7d19b06bfa" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 41349 } } peers { permanent_uuid: "ec8b4c39ed3f419d99b5387ca3e51ca6" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 46215 } } peers { permanent_uuid: "71c25e47472544fa9e042cfcc7bf4f96" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 32929 } }
I20260810 04:07:04.125584 7420 raft_consensus.cc:374] T 7e854480a8364c0195285b9fac067783 P 71c25e47472544fa9e042cfcc7bf4f96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:07:04.125583 7419 raft_consensus.cc:374] T 7e854480a8364c0195285b9fac067783 P 378cfcc378724919ac8f9f7d19b06bfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:07:04.125625 7420 raft_consensus.cc:729] T 7e854480a8364c0195285b9fac067783 P 71c25e47472544fa9e042cfcc7bf4f96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71c25e47472544fa9e042cfcc7bf4f96, State: Initialized, Role: FOLLOWER
I20260810 04:07:04.125625 7419 raft_consensus.cc:729] T 7e854480a8364c0195285b9fac067783 P 378cfcc378724919ac8f9f7d19b06bfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 378cfcc378724919ac8f9f7d19b06bfa, State: Initialized, Role: FOLLOWER
I20260810 04:07:04.125751 7419 consensus_queue.cc:260] T 7e854480a8364c0195285b9fac067783 P 378cfcc378724919ac8f9f7d19b06bfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "378cfcc378724919ac8f9f7d19b06bfa" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 41349 } } peers { permanent_uuid: "ec8b4c39ed3f419d99b5387ca3e51ca6" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 46215 } } peers { permanent_uuid: "71c25e47472544fa9e042cfcc7bf4f96" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 32929 } }
I20260810 04:07:04.125751 7420 consensus_queue.cc:260] T 7e854480a8364c0195285b9fac067783 P 71c25e47472544fa9e042cfcc7bf4f96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "378cfcc378724919ac8f9f7d19b06bfa" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 41349 } } peers { permanent_uuid: "ec8b4c39ed3f419d99b5387ca3e51ca6" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 46215 } } peers { permanent_uuid: "71c25e47472544fa9e042cfcc7bf4f96" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 32929 } }
I20260810 04:07:04.125893 7421 ts_tablet_manager.cc:1428] T 7e854480a8364c0195285b9fac067783 P ec8b4c39ed3f419d99b5387ca3e51ca6: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260810 04:07:04.126004 7089 heartbeater.cc:503] Master 127.31.32.62:35131 was elected leader, sending a full tablet report...
I20260810 04:07:04.126047 7420 ts_tablet_manager.cc:1428] T 7e854480a8364c0195285b9fac067783 P 71c25e47472544fa9e042cfcc7bf4f96: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 04:07:04.126055 7419 ts_tablet_manager.cc:1428] T 7e854480a8364c0195285b9fac067783 P 378cfcc378724919ac8f9f7d19b06bfa: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260810 04:07:04.126495 7390 heartbeater.cc:503] Master 127.31.32.62:35131 was elected leader, sending a full tablet report...
I20260810 04:07:04.126507 7238 heartbeater.cc:503] Master 127.31.32.62:35131 was elected leader, sending a full tablet report...
W20260810 04:07:04.184122 7240 tablet.cc:2369] T 7e854480a8364c0195285b9fac067783 P 378cfcc378724919ac8f9f7d19b06bfa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 04:07:04.278133 7427 raft_consensus.cc:482] T 7e854480a8364c0195285b9fac067783 P 378cfcc378724919ac8f9f7d19b06bfa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 04:07:04.278352 7427 raft_consensus.cc:504] T 7e854480a8364c0195285b9fac067783 P 378cfcc378724919ac8f9f7d19b06bfa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "378cfcc378724919ac8f9f7d19b06bfa" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 41349 } } peers { permanent_uuid: "ec8b4c39ed3f419d99b5387ca3e51ca6" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 46215 } } peers { permanent_uuid: "71c25e47472544fa9e042cfcc7bf4f96" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 32929 } }
I20260810 04:07:04.278784 7427 leader_election.cc:288] T 7e854480a8364c0195285b9fac067783 P 378cfcc378724919ac8f9f7d19b06bfa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ec8b4c39ed3f419d99b5387ca3e51ca6 (127.31.32.1:46215), 71c25e47472544fa9e042cfcc7bf4f96 (127.31.32.3:32929)
W20260810 04:07:04.280972 7092 tablet.cc:2369] T 7e854480a8364c0195285b9fac067783 P ec8b4c39ed3f419d99b5387ca3e51ca6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 04:07:04.282649 7418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:04.279118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.2:43133 (local address 127.31.32.3:32929)
0810 04:07:04.279165 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0810 04:07:04.279168 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:07:04.279182 (+ 14us) server_negotiation.cc:407] Connection header received
0810 04:07:04.279237 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:07:04.279238 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:07:04.279251 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:07:04.279308 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:07:04.279518 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:04.280287 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:04.281706 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:04.282273 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:04.282354 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:04.282362 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0810 04:07:04.282533 (+ 171us) server_negotiation.cc:299] Negotiation successful
0810 04:07:04.282581 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260810 04:07:04.282661 7416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:04.279163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.2:58259 (local address 127.31.32.1:46215)
0810 04:07:04.279206 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0810 04:07:04.279208 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:07:04.279216 (+ 8us) server_negotiation.cc:407] Connection header received
0810 04:07:04.279240 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:07:04.279241 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:07:04.279251 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:07:04.279308 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:07:04.279516 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:04.280297 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:04.281706 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:04.282268 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:04.282350 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:04.282362 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0810 04:07:04.282564 (+ 202us) server_negotiation.cc:299] Negotiation successful
0810 04:07:04.282608 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260810 04:07:04.282668 7244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:04.279050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.1:46215 (local address 127.31.32.2:58259)
0810 04:07:04.279150 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0810 04:07:04.279159 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 04:07:04.279191 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:07:04.279331 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:07:04.279332 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:07:04.279333 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:07:04.279435 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:04.279436 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:04.280381 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:04.280383 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:04.281606 (+ 1223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:04.281609 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:04.282365 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:04.282366 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:04.282450 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:04.282454 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 04:07:04.282526 (+ 72us) client_negotiation.cc:240] Negotiation successful
0810 04:07:04.282559 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89,"mutex_wait_us":144}
I20260810 04:07:04.282694 7243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:04.279024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.3:32929 (local address 127.31.32.2:43133)
0810 04:07:04.279091 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0810 04:07:04.279105 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 04:07:04.279155 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:07:04.279354 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:07:04.279354 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:07:04.279355 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:07:04.279434 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:04.279436 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:04.280388 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:04.280388 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:04.281606 (+ 1218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:04.281609 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:04.282365 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:04.282366 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:04.282450 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:04.282454 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 04:07:04.282513 (+ 59us) client_negotiation.cc:240] Negotiation successful
0810 04:07:04.282559 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45,"mutex_wait_us":17}
I20260810 04:07:04.283077 7038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e854480a8364c0195285b9fac067783" candidate_uuid: "378cfcc378724919ac8f9f7d19b06bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec8b4c39ed3f419d99b5387ca3e51ca6" is_pre_election: true
I20260810 04:07:04.283279 7038 raft_consensus.cc:2454] T 7e854480a8364c0195285b9fac067783 P ec8b4c39ed3f419d99b5387ca3e51ca6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 378cfcc378724919ac8f9f7d19b06bfa in term 0.
I20260810 04:07:04.283285 7337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e854480a8364c0195285b9fac067783" candidate_uuid: "378cfcc378724919ac8f9f7d19b06bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71c25e47472544fa9e042cfcc7bf4f96" is_pre_election: true
I20260810 04:07:04.283486 7337 raft_consensus.cc:2454] T 7e854480a8364c0195285b9fac067783 P 71c25e47472544fa9e042cfcc7bf4f96 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 378cfcc378724919ac8f9f7d19b06bfa in term 0.
I20260810 04:07:04.283530 7115 leader_election.cc:302] T 7e854480a8364c0195285b9fac067783 P 378cfcc378724919ac8f9f7d19b06bfa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 378cfcc378724919ac8f9f7d19b06bfa, ec8b4c39ed3f419d99b5387ca3e51ca6; no voters:
I20260810 04:07:04.283777 7427 raft_consensus.cc:2781] T 7e854480a8364c0195285b9fac067783 P 378cfcc378724919ac8f9f7d19b06bfa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 04:07:04.283830 7427 raft_consensus.cc:482] T 7e854480a8364c0195285b9fac067783 P 378cfcc378724919ac8f9f7d19b06bfa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 04:07:04.283847 7427 raft_consensus.cc:3037] T 7e854480a8364c0195285b9fac067783 P 378cfcc378724919ac8f9f7d19b06bfa [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:07:04.284644 7427 raft_consensus.cc:504] T 7e854480a8364c0195285b9fac067783 P 378cfcc378724919ac8f9f7d19b06bfa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "378cfcc378724919ac8f9f7d19b06bfa" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 41349 } } peers { permanent_uuid: "ec8b4c39ed3f419d99b5387ca3e51ca6" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 46215 } } peers { permanent_uuid: "71c25e47472544fa9e042cfcc7bf4f96" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 32929 } }
I20260810 04:07:04.284796 7427 leader_election.cc:288] T 7e854480a8364c0195285b9fac067783 P 378cfcc378724919ac8f9f7d19b06bfa [CANDIDATE]: Term 1 election: Requested vote from peers ec8b4c39ed3f419d99b5387ca3e51ca6 (127.31.32.1:46215), 71c25e47472544fa9e042cfcc7bf4f96 (127.31.32.3:32929)
I20260810 04:07:04.284953 7038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e854480a8364c0195285b9fac067783" candidate_uuid: "378cfcc378724919ac8f9f7d19b06bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec8b4c39ed3f419d99b5387ca3e51ca6"
I20260810 04:07:04.285058 7038 raft_consensus.cc:3037] T 7e854480a8364c0195285b9fac067783 P ec8b4c39ed3f419d99b5387ca3e51ca6 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:07:04.285351 7337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e854480a8364c0195285b9fac067783" candidate_uuid: "378cfcc378724919ac8f9f7d19b06bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71c25e47472544fa9e042cfcc7bf4f96"
I20260810 04:07:04.285423 7337 raft_consensus.cc:3037] T 7e854480a8364c0195285b9fac067783 P 71c25e47472544fa9e042cfcc7bf4f96 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:07:04.286091 7337 raft_consensus.cc:2454] T 7e854480a8364c0195285b9fac067783 P 71c25e47472544fa9e042cfcc7bf4f96 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 378cfcc378724919ac8f9f7d19b06bfa in term 1.
I20260810 04:07:04.286276 7117 leader_election.cc:302] T 7e854480a8364c0195285b9fac067783 P 378cfcc378724919ac8f9f7d19b06bfa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 378cfcc378724919ac8f9f7d19b06bfa, 71c25e47472544fa9e042cfcc7bf4f96; no voters:
I20260810 04:07:04.286312 7038 raft_consensus.cc:2454] T 7e854480a8364c0195285b9fac067783 P ec8b4c39ed3f419d99b5387ca3e51ca6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 378cfcc378724919ac8f9f7d19b06bfa in term 1.
I20260810 04:07:04.286382 7427 raft_consensus.cc:2781] T 7e854480a8364c0195285b9fac067783 P 378cfcc378724919ac8f9f7d19b06bfa [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:07:04.286590 7427 raft_consensus.cc:686] T 7e854480a8364c0195285b9fac067783 P 378cfcc378724919ac8f9f7d19b06bfa [term 1 LEADER]: Becoming Leader. State: Replica: 378cfcc378724919ac8f9f7d19b06bfa, State: Running, Role: LEADER
I20260810 04:07:04.286798 7427 consensus_queue.cc:237] T 7e854480a8364c0195285b9fac067783 P 378cfcc378724919ac8f9f7d19b06bfa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "378cfcc378724919ac8f9f7d19b06bfa" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 41349 } } peers { permanent_uuid: "ec8b4c39ed3f419d99b5387ca3e51ca6" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 46215 } } peers { permanent_uuid: "71c25e47472544fa9e042cfcc7bf4f96" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 32929 } }
I20260810 04:07:04.287693 6027 catalog_manager.cc:5495] T 7e854480a8364c0195285b9fac067783 P 378cfcc378724919ac8f9f7d19b06bfa reported cstate change: term changed from 0 to 1, leader changed from <none> to 378cfcc378724919ac8f9f7d19b06bfa (127.31.32.2). New cstate: current_term: 1 leader_uuid: "378cfcc378724919ac8f9f7d19b06bfa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "378cfcc378724919ac8f9f7d19b06bfa" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 41349 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec8b4c39ed3f419d99b5387ca3e51ca6" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 46215 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "71c25e47472544fa9e042cfcc7bf4f96" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 32929 } health_report { overall_health: UNKNOWN } } }
I20260810 04:07:04.296587 7415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:04.294155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54570 (local address 127.31.32.2:41349)
0810 04:07:04.294211 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0810 04:07:04.294213 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:07:04.294222 (+ 9us) server_negotiation.cc:407] Connection header received
0810 04:07:04.294266 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:07:04.294267 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:07:04.294285 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:07:04.294353 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 04:07:04.294519 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:04.294982 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:04.295745 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:04.295954 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:04.296068 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:04.296242 (+ 174us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 04:07:04.296346 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 04:07:04.296471 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0810 04:07:04.296490 (+ 19us) server_negotiation.cc:299] Negotiation successful
0810 04:07:04.296512 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260810 04:07:04.298272 7337 raft_consensus.cc:1261] T 7e854480a8364c0195285b9fac067783 P 71c25e47472544fa9e042cfcc7bf4f96 [term 1 FOLLOWER]: Refusing update from remote peer 378cfcc378724919ac8f9f7d19b06bfa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:07:04.298367 7038 raft_consensus.cc:1261] T 7e854480a8364c0195285b9fac067783 P ec8b4c39ed3f419d99b5387ca3e51ca6 [term 1 FOLLOWER]: Refusing update from remote peer 378cfcc378724919ac8f9f7d19b06bfa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:07:04.298645 7428 consensus_queue.cc:1035] T 7e854480a8364c0195285b9fac067783 P 378cfcc378724919ac8f9f7d19b06bfa [LEADER]: Connected to new peer: Peer: permanent_uuid: "71c25e47472544fa9e042cfcc7bf4f96" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 32929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:07:04.298753 7427 consensus_queue.cc:1035] T 7e854480a8364c0195285b9fac067783 P 378cfcc378724919ac8f9f7d19b06bfa [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec8b4c39ed3f419d99b5387ca3e51ca6" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 46215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:07:04.301229 7433 mvcc.cc:205] Tried to move back new op lower bound from 7316827440323227648 to 7316827440279293952. Current Snapshot: MvccSnapshot[applied={T|T < 7316827440323227648}]
I20260810 04:07:04.301630 7434 mvcc.cc:205] Tried to move back new op lower bound from 7316827440323227648 to 7316827440279293952. Current Snapshot: MvccSnapshot[applied={T|T < 7316827440323227648}]
I20260810 04:07:04.303826 6027 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50176:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 04:07:04.303922 6027 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50176:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 04:07:04.304045 7436 mvcc.cc:205] Tried to move back new op lower bound from 7316827440323227648 to 7316827440279293952. Current Snapshot: MvccSnapshot[applied={T|T < 7316827440323227648}]
I20260810 04:07:04.305361 6027 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b858c2bd3c2346bf9ac073d925ee5ef1: Sending DeleteTablet for 3 replicas of tablet 7e854480a8364c0195285b9fac067783
I20260810 04:07:04.305657 7164 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e854480a8364c0195285b9fac067783 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:07:04 UTC) from {username='oryx', principal='oryx/127.31.32.62@KRBTEST.COM'} at 127.0.0.1:54562
I20260810 04:07:04.305703 7315 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e854480a8364c0195285b9fac067783 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:07:04 UTC) from {username='oryx', principal='oryx/127.31.32.62@KRBTEST.COM'} at 127.0.0.1:38470
I20260810 04:07:04.305779 7016 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e854480a8364c0195285b9fac067783 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:07:04 UTC) from {username='oryx', principal='oryx/127.31.32.62@KRBTEST.COM'} at 127.0.0.1:35818
I20260810 04:07:04.305968 7445 tablet_replica.cc:330] T 7e854480a8364c0195285b9fac067783 P 71c25e47472544fa9e042cfcc7bf4f96: stopping tablet replica
I20260810 04:07:04.306077 7445 raft_consensus.cc:2229] T 7e854480a8364c0195285b9fac067783 P 71c25e47472544fa9e042cfcc7bf4f96 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 04:07:04.306185 7445 raft_consensus.cc:2258] T 7e854480a8364c0195285b9fac067783 P 71c25e47472544fa9e042cfcc7bf4f96 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:07:04.306216 7446 tablet_replica.cc:330] T 7e854480a8364c0195285b9fac067783 P ec8b4c39ed3f419d99b5387ca3e51ca6: stopping tablet replica
I20260810 04:07:04.306514 7446 raft_consensus.cc:2229] T 7e854480a8364c0195285b9fac067783 P ec8b4c39ed3f419d99b5387ca3e51ca6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 04:07:04.306671 7446 raft_consensus.cc:2258] T 7e854480a8364c0195285b9fac067783 P ec8b4c39ed3f419d99b5387ca3e51ca6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:07:04.306970 7444 tablet_replica.cc:330] T 7e854480a8364c0195285b9fac067783 P 378cfcc378724919ac8f9f7d19b06bfa: stopping tablet replica
I20260810 04:07:04.307108 7444 raft_consensus.cc:2229] T 7e854480a8364c0195285b9fac067783 P 378cfcc378724919ac8f9f7d19b06bfa [term 1 LEADER]: Raft consensus shutting down.
I20260810 04:07:04.307163 31872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskws89Io/build/release/bin/kudu with pid 5991
I20260810 04:07:04.307319 7444 raft_consensus.cc:2258] T 7e854480a8364c0195285b9fac067783 P 378cfcc378724919ac8f9f7d19b06bfa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:07:04.309381 7444 ts_tablet_manager.cc:1905] T 7e854480a8364c0195285b9fac067783 P 378cfcc378724919ac8f9f7d19b06bfa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:07:04.311658 7444 ts_tablet_manager.cc:1918] T 7e854480a8364c0195285b9fac067783 P 378cfcc378724919ac8f9f7d19b06bfa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 04:07:04.311798 7444 log.cc:1196] T 7e854480a8364c0195285b9fac067783 P 378cfcc378724919ac8f9f7d19b06bfa: Deleting WAL directory at /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/minicluster-data/ts-1/wal/wals/7e854480a8364c0195285b9fac067783
I20260810 04:07:04.312215 7444 ts_tablet_manager.cc:1939] T 7e854480a8364c0195285b9fac067783 P 378cfcc378724919ac8f9f7d19b06bfa: Deleting consensus metadata
I20260810 04:07:04.316402 7445 ts_tablet_manager.cc:1905] T 7e854480a8364c0195285b9fac067783 P 71c25e47472544fa9e042cfcc7bf4f96: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:07:04.316527 7406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:04.306423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53370 (local address 127.31.32.60:33905)
0810 04:07:04.306525 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0810 04:07:04.306527 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:07:04.306579 (+ 52us) server_negotiation.cc:407] Connection header received
0810 04:07:04.306671 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:07:04.306672 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:07:04.306683 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:07:04.306713 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:07:04.308522 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:04.309545 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:04.310865 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:04.311237 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:04.311283 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:04.313311 (+ 2028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:07:04.313313 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:07:04.313313 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:07:04.313322 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:07:04.315183 (+ 1861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:07:04.315618 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:07:04.315619 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:07:04.315620 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:07:04.315688 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:07:04.315915 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:07:04.315917 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:07:04.315917 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:07:04.316080 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:07:04.316224 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0810 04:07:04.316340 (+ 116us) server_negotiation.cc:299] Negotiation successful
0810 04:07:04.316437 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260810 04:07:04.318177 7445 ts_tablet_manager.cc:1918] T 7e854480a8364c0195285b9fac067783 P 71c25e47472544fa9e042cfcc7bf4f96: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 04:07:04.318281 7445 log.cc:1196] T 7e854480a8364c0195285b9fac067783 P 71c25e47472544fa9e042cfcc7bf4f96: Deleting WAL directory at /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/minicluster-data/ts-2/wal/wals/7e854480a8364c0195285b9fac067783
I20260810 04:07:04.318604 7445 ts_tablet_manager.cc:1939] T 7e854480a8364c0195285b9fac067783 P 71c25e47472544fa9e042cfcc7bf4f96: Deleting consensus metadata
W20260810 04:07:04.318778 7114 connection.cc:535] server connection from 127.0.0.1:54562 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260810 04:07:04.319052 7263 connection.cc:441] server connection from 127.0.0.1:38470 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38470 (request call id 1) could send its response
I20260810 04:07:04.319993 7446 ts_tablet_manager.cc:1905] T 7e854480a8364c0195285b9fac067783 P ec8b4c39ed3f419d99b5387ca3e51ca6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:07:04.321694 7446 ts_tablet_manager.cc:1918] T 7e854480a8364c0195285b9fac067783 P ec8b4c39ed3f419d99b5387ca3e51ca6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 04:07:04.321763 7446 log.cc:1196] T 7e854480a8364c0195285b9fac067783 P ec8b4c39ed3f419d99b5387ca3e51ca6: Deleting WAL directory at /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/minicluster-data/ts-0/wal/wals/7e854480a8364c0195285b9fac067783
I20260810 04:07:04.322026 7446 ts_tablet_manager.cc:1939] T 7e854480a8364c0195285b9fac067783 P ec8b4c39ed3f419d99b5387ca3e51ca6: Deleting consensus metadata
W20260810 04:07:04.322432 6966 connection.cc:441] server connection from 127.0.0.1:35818 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35818 (request call id 1) could send its response
I20260810 04:07:04.330101 6745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:04.322874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34012 (local address 127.31.32.61:40815)
0810 04:07:04.322935 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0810 04:07:04.322937 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:07:04.323104 (+ 167us) server_negotiation.cc:407] Connection header received
0810 04:07:04.323142 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:07:04.323143 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:07:04.323156 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:07:04.323222 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:07:04.324081 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:04.324633 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:04.325385 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:04.325579 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:04.325654 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:04.326992 (+ 1338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:07:04.326993 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:07:04.326993 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:07:04.327000 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:07:04.328785 (+ 1785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:07:04.329162 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:07:04.329163 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:07:04.329164 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:07:04.329218 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:07:04.329533 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:07:04.329534 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:07:04.329534 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:07:04.329653 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:07:04.329739 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0810 04:07:04.329941 (+ 202us) server_negotiation.cc:299] Negotiation successful
0810 04:07:04.330025 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260810 04:07:04.330473 7406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:04.320818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53374 (local address 127.31.32.60:33905)
0810 04:07:04.322521 (+ 1703us) server_negotiation.cc:206] Beginning negotiation
0810 04:07:04.322523 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:07:04.322533 (+ 10us) server_negotiation.cc:407] Connection header received
0810 04:07:04.322557 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:07:04.322558 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:07:04.322569 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:07:04.322638 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:07:04.323399 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:04.324321 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:04.325524 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:04.325853 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:04.325894 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:04.327410 (+ 1516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:07:04.327412 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:07:04.327412 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:07:04.327419 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:07:04.329090 (+ 1671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:07:04.329536 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:07:04.329537 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:07:04.329537 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:07:04.329594 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:07:04.329838 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:07:04.329839 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:07:04.329839 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:07:04.329980 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:07:04.330049 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0810 04:07:04.330293 (+ 244us) server_negotiation.cc:299] Negotiation successful
0810 04:07:04.330371 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1680}
I20260810 04:07:05.303427 7454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:05.303189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:35131 (local address 127.31.32.1:60473)
0810 04:07:05.303303 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0810 04:07:05.303378 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.62:35131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":55,"threads_started":1}
W20260810 04:07:05.303730 7089 heartbeater.cc:644] Failed to heartbeat to 127.31.32.62:35131 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.31.32.62:35131: connect: Connection refused (error 111)
I20260810 04:07:05.304023 7455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:05.303673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:35131 (local address 127.31.32.2:33085)
0810 04:07:05.303894 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0810 04:07:05.303981 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.62:35131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":61,"threads_started":1}
W20260810 04:07:05.304237 7238 heartbeater.cc:644] Failed to heartbeat to 127.31.32.62:35131 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.31.32.62:35131: connect: Connection refused (error 111)
I20260810 04:07:05.306437 7456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:05.306208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:35131 (local address 127.31.32.3:59133)
0810 04:07:05.306325 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0810 04:07:05.306396 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.62:35131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":47,"threads_started":1}
W20260810 04:07:05.306725 7390 heartbeater.cc:644] Failed to heartbeat to 127.31.32.62:35131 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.31.32.62:35131: connect: Connection refused (error 111)
I20260810 04:07:05.805855 7457 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8b612a0b090e4e71a02370b642ea30ab [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b858c2bd3c2346bf9ac073d925ee5ef1)
I20260810 04:07:05.806044 7457 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8b612a0b090e4e71a02370b642ea30ab [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b858c2bd3c2346bf9ac073d925ee5ef1" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 35131 } } peers { permanent_uuid: "2138c4c9465e4cd19eeef783497f22e5" member_type: VOTER last_known_addr { host: "127.31.32.61" port: 40815 } attrs { promote: false } } peers { permanent_uuid: "8b612a0b090e4e71a02370b642ea30ab" member_type: VOTER last_known_addr { host: "127.31.32.60" port: 33905 } attrs { promote: false } }
I20260810 04:07:05.806273 7458 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2138c4c9465e4cd19eeef783497f22e5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b858c2bd3c2346bf9ac073d925ee5ef1)
I20260810 04:07:05.806351 7457 leader_election.cc:288] T 00000000000000000000000000000000 P 8b612a0b090e4e71a02370b642ea30ab [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b858c2bd3c2346bf9ac073d925ee5ef1 (127.31.32.62:35131), 2138c4c9465e4cd19eeef783497f22e5 (127.31.32.61:40815)
I20260810 04:07:05.806414 7458 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2138c4c9465e4cd19eeef783497f22e5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b858c2bd3c2346bf9ac073d925ee5ef1" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 35131 } } peers { permanent_uuid: "2138c4c9465e4cd19eeef783497f22e5" member_type: VOTER last_known_addr { host: "127.31.32.61" port: 40815 } attrs { promote: false } } peers { permanent_uuid: "8b612a0b090e4e71a02370b642ea30ab" member_type: VOTER last_known_addr { host: "127.31.32.60" port: 33905 } attrs { promote: false } }
I20260810 04:07:05.806680 7459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:05.806478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:35131 (local address 127.0.0.1:50602)
0810 04:07:05.806587 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0810 04:07:05.806637 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.62:35131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":46,"threads_started":1}
W20260810 04:07:05.806805 6874 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.32.62:35131: connect: Connection refused (error 111)
I20260810 04:07:05.807193 7459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:05.807108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:35131 (local address 127.0.0.1:50626)
0810 04:07:05.807143 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0810 04:07:05.807170 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.62:35131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
W20260810 04:07:05.807300 6874 leader_election.cc:334] T 00000000000000000000000000000000 P 8b612a0b090e4e71a02370b642ea30ab [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b858c2bd3c2346bf9ac073d925ee5ef1 (127.31.32.62:35131): Network error: Client connection negotiation failed: client connection to 127.31.32.62:35131: connect: Connection refused (error 111)
I20260810 04:07:05.807114 7462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:05.806866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:35131 (local address 127.0.0.1:50614)
0810 04:07:05.807021 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0810 04:07:05.807076 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.62:35131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":45,"threads_started":1}
I20260810 04:07:05.807204 7458 leader_election.cc:288] T 00000000000000000000000000000000 P 2138c4c9465e4cd19eeef783497f22e5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b858c2bd3c2346bf9ac073d925ee5ef1 (127.31.32.62:35131), 8b612a0b090e4e71a02370b642ea30ab (127.31.32.60:33905)
W20260810 04:07:05.807478 6677 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.32.62:35131: connect: Connection refused (error 111)
I20260810 04:07:05.807919 7462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:05.807801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:35131 (local address 127.0.0.1:50632)
0810 04:07:05.807860 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0810 04:07:05.807893 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.62:35131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260810 04:07:05.808063 6677 leader_election.cc:334] T 00000000000000000000000000000000 P 2138c4c9465e4cd19eeef783497f22e5 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b858c2bd3c2346bf9ac073d925ee5ef1 (127.31.32.62:35131): Network error: Client connection negotiation failed: client connection to 127.31.32.62:35131: connect: Connection refused (error 111)
I20260810 04:07:05.809893 7460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:05.806520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.61:40815 (local address 127.0.0.1:34026)
0810 04:07:05.806677 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0810 04:07:05.806688 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 04:07:05.806758 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:07:05.807040 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:07:05.807041 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:07:05.807044 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:07:05.807105 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:05.807107 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:05.808237 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:05.808238 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:05.809032 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:05.809035 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:05.809630 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:05.809631 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:05.809695 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:05.809700 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 04:07:05.809752 (+ 52us) client_negotiation.cc:240] Negotiation successful
0810 04:07:05.809791 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":48,"threads_started":1}
I20260810 04:07:05.809897 7461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:05.806567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34026 (local address 127.31.32.61:40815)
0810 04:07:05.806871 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0810 04:07:05.806873 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:07:05.806882 (+ 9us) server_negotiation.cc:407] Connection header received
0810 04:07:05.806909 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:07:05.806910 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:07:05.806920 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:07:05.806982 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:07:05.807183 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:05.808156 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:05.809115 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:05.809555 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:05.809608 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:05.809617 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0810 04:07:05.809763 (+ 146us) server_negotiation.cc:299] Negotiation successful
0810 04:07:05.809806 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":40,"threads_started":1}
I20260810 04:07:05.809988 7464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:05.807052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.60:33905 (local address 127.0.0.1:53378)
0810 04:07:05.807236 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0810 04:07:05.807245 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 04:07:05.807316 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:07:05.807508 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:07:05.807509 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:07:05.807511 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:07:05.807552 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:05.807554 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:05.808410 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:05.808411 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:05.809168 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:05.809170 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:05.809709 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:05.809710 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:05.809790 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:05.809792 (+ 2us) client_negotiation.cc:769] Sending connection context
0810 04:07:05.809848 (+ 56us) client_negotiation.cc:240] Negotiation successful
0810 04:07:05.809881 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":33,"threads_started":1}
I20260810 04:07:05.810190 6701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8b612a0b090e4e71a02370b642ea30ab" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "2138c4c9465e4cd19eeef783497f22e5" is_pre_election: true
I20260810 04:07:05.810333 6701 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2138c4c9465e4cd19eeef783497f22e5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b612a0b090e4e71a02370b642ea30ab in term 1.
I20260810 04:07:05.810460 7465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:05.807207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53378 (local address 127.31.32.60:33905)
0810 04:07:05.807357 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0810 04:07:05.807359 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:07:05.807367 (+ 8us) server_negotiation.cc:407] Connection header received
0810 04:07:05.807390 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:07:05.807390 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:07:05.807400 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:07:05.807460 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:07:05.807606 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:05.808334 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:05.809246 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:05.809639 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:05.809700 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:05.809710 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0810 04:07:05.809864 (+ 154us) server_negotiation.cc:299] Negotiation successful
0810 04:07:05.809899 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":129,"threads_started":1}
I20260810 04:07:05.810479 6873 leader_election.cc:302] T 00000000000000000000000000000000 P 8b612a0b090e4e71a02370b642ea30ab [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2138c4c9465e4cd19eeef783497f22e5, 8b612a0b090e4e71a02370b642ea30ab; no voters: b858c2bd3c2346bf9ac073d925ee5ef1
I20260810 04:07:05.810554 7457 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8b612a0b090e4e71a02370b642ea30ab [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260810 04:07:05.810585 7457 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8b612a0b090e4e71a02370b642ea30ab [term 1 FOLLOWER]: Starting leader election (detected failure of leader b858c2bd3c2346bf9ac073d925ee5ef1)
I20260810 04:07:05.810606 7457 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8b612a0b090e4e71a02370b642ea30ab [term 1 FOLLOWER]: Advancing to term 2
I20260810 04:07:05.810738 6905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2138c4c9465e4cd19eeef783497f22e5" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "8b612a0b090e4e71a02370b642ea30ab" is_pre_election: true
I20260810 04:07:05.811554 7457 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8b612a0b090e4e71a02370b642ea30ab [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b858c2bd3c2346bf9ac073d925ee5ef1" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 35131 } } peers { permanent_uuid: "2138c4c9465e4cd19eeef783497f22e5" member_type: VOTER last_known_addr { host: "127.31.32.61" port: 40815 } attrs { promote: false } } peers { permanent_uuid: "8b612a0b090e4e71a02370b642ea30ab" member_type: VOTER last_known_addr { host: "127.31.32.60" port: 33905 } attrs { promote: false } }
I20260810 04:07:05.811700 7457 leader_election.cc:288] T 00000000000000000000000000000000 P 8b612a0b090e4e71a02370b642ea30ab [CANDIDATE]: Term 2 election: Requested vote from peers b858c2bd3c2346bf9ac073d925ee5ef1 (127.31.32.62:35131), 2138c4c9465e4cd19eeef783497f22e5 (127.31.32.61:40815)
I20260810 04:07:05.811744 6905 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 8b612a0b090e4e71a02370b642ea30ab [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2138c4c9465e4cd19eeef783497f22e5 in current term 2: Already voted for candidate 8b612a0b090e4e71a02370b642ea30ab in this term.
I20260810 04:07:05.811873 6701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8b612a0b090e4e71a02370b642ea30ab" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "2138c4c9465e4cd19eeef783497f22e5"
I20260810 04:07:05.811931 7460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:05.811767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:35131 (local address 127.0.0.1:50642)
0810 04:07:05.811868 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0810 04:07:05.811904 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.62:35131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260810 04:07:05.811962 6701 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2138c4c9465e4cd19eeef783497f22e5 [term 1 FOLLOWER]: Advancing to term 2
I20260810 04:07:05.812302 6677 leader_election.cc:302] T 00000000000000000000000000000000 P 2138c4c9465e4cd19eeef783497f22e5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2138c4c9465e4cd19eeef783497f22e5; no voters: 8b612a0b090e4e71a02370b642ea30ab, b858c2bd3c2346bf9ac073d925ee5ef1
I20260810 04:07:05.812736 6701 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2138c4c9465e4cd19eeef783497f22e5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b612a0b090e4e71a02370b642ea30ab in term 2.
I20260810 04:07:05.812860 7458 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 2138c4c9465e4cd19eeef783497f22e5 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 04:07:05.812911 6873 leader_election.cc:302] T 00000000000000000000000000000000 P 8b612a0b090e4e71a02370b642ea30ab [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2138c4c9465e4cd19eeef783497f22e5, 8b612a0b090e4e71a02370b642ea30ab; no voters:
I20260810 04:07:05.813011 7457 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8b612a0b090e4e71a02370b642ea30ab [term 2 FOLLOWER]: Leader election won for term 2
I20260810 04:07:05.813160 7457 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8b612a0b090e4e71a02370b642ea30ab [term 2 LEADER]: Becoming Leader. State: Replica: 8b612a0b090e4e71a02370b642ea30ab, State: Running, Role: LEADER
I20260810 04:07:05.813210 7460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:05.813094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:35131 (local address 127.0.0.1:50654)
0810 04:07:05.813137 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0810 04:07:05.813161 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.62:35131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260810 04:07:05.813261 6874 leader_election.cc:334] T 00000000000000000000000000000000 P 8b612a0b090e4e71a02370b642ea30ab [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer b858c2bd3c2346bf9ac073d925ee5ef1 (127.31.32.62:35131): Network error: Client connection negotiation failed: client connection to 127.31.32.62:35131: connect: Connection refused (error 111)
I20260810 04:07:05.813249 7457 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8b612a0b090e4e71a02370b642ea30ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b858c2bd3c2346bf9ac073d925ee5ef1" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 35131 } } peers { permanent_uuid: "2138c4c9465e4cd19eeef783497f22e5" member_type: VOTER last_known_addr { host: "127.31.32.61" port: 40815 } attrs { promote: false } } peers { permanent_uuid: "8b612a0b090e4e71a02370b642ea30ab" member_type: VOTER last_known_addr { host: "127.31.32.60" port: 33905 } attrs { promote: false } }
I20260810 04:07:05.813628 7467 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b612a0b090e4e71a02370b642ea30ab [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8b612a0b090e4e71a02370b642ea30ab. Latest consensus state: current_term: 2 leader_uuid: "8b612a0b090e4e71a02370b642ea30ab" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b858c2bd3c2346bf9ac073d925ee5ef1" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 35131 } } peers { permanent_uuid: "2138c4c9465e4cd19eeef783497f22e5" member_type: VOTER last_known_addr { host: "127.31.32.61" port: 40815 } attrs { promote: false } } peers { permanent_uuid: "8b612a0b090e4e71a02370b642ea30ab" member_type: VOTER last_known_addr { host: "127.31.32.60" port: 33905 } attrs { promote: false } } }
I20260810 04:07:05.813727 7467 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b612a0b090e4e71a02370b642ea30ab [sys.catalog]: This master's current role is: LEADER
I20260810 04:07:05.813912 7470 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 04:07:05.814365 7470 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 04:07:05.814424 7470 catalog_manager.cc:1259] Loaded cluster ID: 1dd3f94c8dd942a4acce7f7524739ff2
I20260810 04:07:05.814440 7470 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 04:07:05.814551 7470 catalog_manager.cc:1504] Loading token signing keys...
I20260810 04:07:05.814599 7470 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 8b612a0b090e4e71a02370b642ea30ab: Loaded TSK: 0
I20260810 04:07:05.814702 7470 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 04:07:05.871745 6890 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53374:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 04:07:05.873342 6890 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 04:07:05.874285 7460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:05.874175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:35131 (local address 127.0.0.1:50658)
0810 04:07:05.874227 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0810 04:07:05.874260 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.62:35131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260810 04:07:05.874280 6701 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2138c4c9465e4cd19eeef783497f22e5 [term 2 FOLLOWER]: Refusing update from remote peer 8b612a0b090e4e71a02370b642ea30ab: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260810 04:07:05.874538 7467 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8b612a0b090e4e71a02370b642ea30ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "2138c4c9465e4cd19eeef783497f22e5" member_type: VOTER last_known_addr { host: "127.31.32.61" port: 40815 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260810 04:07:05.874714 7460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:05.874592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:35131 (local address 127.0.0.1:50674)
0810 04:07:05.874656 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0810 04:07:05.874688 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.62:35131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260810 04:07:05.874787 6874 consensus_peers.cc:487] T 00000000000000000000000000000000 P 8b612a0b090e4e71a02370b642ea30ab -> Peer b858c2bd3c2346bf9ac073d925ee5ef1 (127.31.32.62:35131): Couldn't send request to peer b858c2bd3c2346bf9ac073d925ee5ef1. Status: Network error: Client connection negotiation failed: client connection to 127.31.32.62:35131: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260810 04:07:05.875149 7458 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2138c4c9465e4cd19eeef783497f22e5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8b612a0b090e4e71a02370b642ea30ab. Latest consensus state: current_term: 2 leader_uuid: "8b612a0b090e4e71a02370b642ea30ab" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b858c2bd3c2346bf9ac073d925ee5ef1" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 35131 } } peers { permanent_uuid: "2138c4c9465e4cd19eeef783497f22e5" member_type: VOTER last_known_addr { host: "127.31.32.61" port: 40815 } attrs { promote: false } } peers { permanent_uuid: "8b612a0b090e4e71a02370b642ea30ab" member_type: VOTER last_known_addr { host: "127.31.32.60" port: 33905 } attrs { promote: false } } }
I20260810 04:07:05.875274 7458 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2138c4c9465e4cd19eeef783497f22e5 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 04:07:05.875751 7457 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b612a0b090e4e71a02370b642ea30ab [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "8b612a0b090e4e71a02370b642ea30ab" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b858c2bd3c2346bf9ac073d925ee5ef1" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 35131 } } peers { permanent_uuid: "2138c4c9465e4cd19eeef783497f22e5" member_type: VOTER last_known_addr { host: "127.31.32.61" port: 40815 } attrs { promote: false } } peers { permanent_uuid: "8b612a0b090e4e71a02370b642ea30ab" member_type: VOTER last_known_addr { host: "127.31.32.60" port: 33905 } attrs { promote: false } } }
I20260810 04:07:05.875847 7457 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b612a0b090e4e71a02370b642ea30ab [sys.catalog]: This master's current role is: LEADER
I20260810 04:07:05.876174 7458 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2138c4c9465e4cd19eeef783497f22e5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "8b612a0b090e4e71a02370b642ea30ab" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b858c2bd3c2346bf9ac073d925ee5ef1" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 35131 } } peers { permanent_uuid: "2138c4c9465e4cd19eeef783497f22e5" member_type: VOTER last_known_addr { host: "127.31.32.61" port: 40815 } attrs { promote: false } } peers { permanent_uuid: "8b612a0b090e4e71a02370b642ea30ab" member_type: VOTER last_known_addr { host: "127.31.32.60" port: 33905 } attrs { promote: false } } }
I20260810 04:07:05.876264 7458 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2138c4c9465e4cd19eeef783497f22e5 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 04:07:05.880882 7459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:05.877592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.2:41349 (local address 127.0.0.1:54580)
0810 04:07:05.877712 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0810 04:07:05.877722 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 04:07:05.877770 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:07:05.878030 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:07:05.878031 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:07:05.878034 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:07:05.878104 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:05.878106 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:05.878794 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:05.878795 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:05.879941 (+ 1146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:05.879943 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:05.880616 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:05.880617 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:05.880686 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:05.880689 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 04:07:05.880771 (+ 82us) client_negotiation.cc:240] Negotiation successful
0810 04:07:05.880811 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"mutex_wait_us":2}
I20260810 04:07:05.880892 7477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:05.877643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54580 (local address 127.31.32.2:41349)
0810 04:07:05.877892 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0810 04:07:05.877894 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:07:05.877902 (+ 8us) server_negotiation.cc:407] Connection header received
0810 04:07:05.877930 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:07:05.877931 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:07:05.877944 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:07:05.878163 (+ 219us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:07:05.878203 (+ 40us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:05.878706 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:05.880034 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:05.880535 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:05.880618 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:05.880631 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0810 04:07:05.880780 (+ 149us) server_negotiation.cc:299] Negotiation successful
0810 04:07:05.880826 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":228,"thread_start_us":58,"threads_started":1}
I20260810 04:07:05.881021 7479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:05.877713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35820 (local address 127.31.32.1:46215)
0810 04:07:05.877815 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0810 04:07:05.877817 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:07:05.877973 (+ 156us) server_negotiation.cc:407] Connection header received
0810 04:07:05.878065 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:07:05.878066 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:07:05.878078 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:07:05.878115 (+ 37us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:07:05.878283 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:05.879006 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:05.880046 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:05.880640 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:05.880688 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:05.880698 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0810 04:07:05.880901 (+ 203us) server_negotiation.cc:299] Negotiation successful
0810 04:07:05.880940 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81,"thread_start_us":51,"threads_started":1}
I20260810 04:07:05.881114 7460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:05.877624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.3:32929 (local address 127.0.0.1:38472)
0810 04:07:05.877732 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0810 04:07:05.877743 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 04:07:05.877785 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:07:05.878057 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:07:05.878057 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:07:05.878059 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:07:05.878103 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:05.878105 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:05.879069 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:05.879070 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:05.880208 (+ 1138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:05.880210 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:05.880922 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:05.880923 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:05.880969 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:05.880973 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 04:07:05.881001 (+ 28us) client_negotiation.cc:240] Negotiation successful
0810 04:07:05.881039 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93,"mutex_wait_us":70}
I20260810 04:07:05.881282 7478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:05.877650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.1:46215 (local address 127.0.0.1:35820)
0810 04:07:05.877916 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0810 04:07:05.877925 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 04:07:05.877997 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:07:05.878176 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:07:05.878176 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:07:05.878178 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:07:05.878211 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:05.878212 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:05.879104 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:05.879105 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:05.879944 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:05.879947 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:05.880757 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:05.880757 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:05.880816 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:05.880819 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 04:07:05.881142 (+ 323us) client_negotiation.cc:240] Negotiation successful
0810 04:07:05.881189 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"mutex_wait_us":26,"thread_start_us":81,"threads_started":1}
I20260810 04:07:05.881422 7164 tablet_service.cc:1467] Processing CreateTablet for tablet c6b6bd3255d14a0cb254e779b8bde15a (DEFAULT_TABLE table=test-table [id=82c417f159f447a08dfbd6f5262e45b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:07:05.881493 7016 tablet_service.cc:1467] Processing CreateTablet for tablet c6b6bd3255d14a0cb254e779b8bde15a (DEFAULT_TABLE table=test-table [id=82c417f159f447a08dfbd6f5262e45b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:07:05.881528 7480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:05.877812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38472 (local address 127.31.32.3:32929)
0810 04:07:05.877927 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0810 04:07:05.877929 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:07:05.877936 (+ 7us) server_negotiation.cc:407] Connection header received
0810 04:07:05.877962 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:07:05.877963 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:07:05.877972 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:07:05.878163 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:07:05.878206 (+ 43us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:05.878966 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:05.880283 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:05.880824 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:05.881319 (+ 495us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:05.881334 (+ 15us) server_negotiation.cc:1035] Waiting for connection context
0810 04:07:05.881420 (+ 86us) server_negotiation.cc:299] Negotiation successful
0810 04:07:05.881464 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94,"thread_start_us":51,"threads_started":1}
I20260810 04:07:05.881556 7164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6b6bd3255d14a0cb254e779b8bde15a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:07:05.881623 7016 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6b6bd3255d14a0cb254e779b8bde15a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:07:05.881675 7315 tablet_service.cc:1467] Processing CreateTablet for tablet c6b6bd3255d14a0cb254e779b8bde15a (DEFAULT_TABLE table=test-table [id=82c417f159f447a08dfbd6f5262e45b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:07:05.881778 7315 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6b6bd3255d14a0cb254e779b8bde15a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:07:05.882810 7481 tablet_bootstrap.cc:492] T c6b6bd3255d14a0cb254e779b8bde15a P 378cfcc378724919ac8f9f7d19b06bfa: Bootstrap starting.
I20260810 04:07:05.883061 7482 tablet_bootstrap.cc:492] T c6b6bd3255d14a0cb254e779b8bde15a P ec8b4c39ed3f419d99b5387ca3e51ca6: Bootstrap starting.
I20260810 04:07:05.883226 7483 tablet_bootstrap.cc:492] T c6b6bd3255d14a0cb254e779b8bde15a P 71c25e47472544fa9e042cfcc7bf4f96: Bootstrap starting.
I20260810 04:07:05.883369 7482 tablet_bootstrap.cc:654] T c6b6bd3255d14a0cb254e779b8bde15a P ec8b4c39ed3f419d99b5387ca3e51ca6: Neither blocks nor log segments found. Creating new log.
I20260810 04:07:05.883383 7481 tablet_bootstrap.cc:654] T c6b6bd3255d14a0cb254e779b8bde15a P 378cfcc378724919ac8f9f7d19b06bfa: Neither blocks nor log segments found. Creating new log.
I20260810 04:07:05.883615 7483 tablet_bootstrap.cc:654] T c6b6bd3255d14a0cb254e779b8bde15a P 71c25e47472544fa9e042cfcc7bf4f96: Neither blocks nor log segments found. Creating new log.
I20260810 04:07:05.884032 7482 tablet_bootstrap.cc:492] T c6b6bd3255d14a0cb254e779b8bde15a P ec8b4c39ed3f419d99b5387ca3e51ca6: No bootstrap required, opened a new log
I20260810 04:07:05.884081 7482 ts_tablet_manager.cc:1397] T c6b6bd3255d14a0cb254e779b8bde15a P ec8b4c39ed3f419d99b5387ca3e51ca6: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260810 04:07:05.884122 7481 tablet_bootstrap.cc:492] T c6b6bd3255d14a0cb254e779b8bde15a P 378cfcc378724919ac8f9f7d19b06bfa: No bootstrap required, opened a new log
I20260810 04:07:05.884262 7481 ts_tablet_manager.cc:1397] T c6b6bd3255d14a0cb254e779b8bde15a P 378cfcc378724919ac8f9f7d19b06bfa: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260810 04:07:05.884274 7482 raft_consensus.cc:348] T c6b6bd3255d14a0cb254e779b8bde15a P ec8b4c39ed3f419d99b5387ca3e51ca6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "378cfcc378724919ac8f9f7d19b06bfa" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 41349 } } peers { permanent_uuid: "71c25e47472544fa9e042cfcc7bf4f96" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 32929 } } peers { permanent_uuid: "ec8b4c39ed3f419d99b5387ca3e51ca6" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 46215 } }
I20260810 04:07:05.884353 7482 raft_consensus.cc:374] T c6b6bd3255d14a0cb254e779b8bde15a P ec8b4c39ed3f419d99b5387ca3e51ca6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:07:05.884379 7482 raft_consensus.cc:729] T c6b6bd3255d14a0cb254e779b8bde15a P ec8b4c39ed3f419d99b5387ca3e51ca6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec8b4c39ed3f419d99b5387ca3e51ca6, State: Initialized, Role: FOLLOWER
I20260810 04:07:05.884378 7481 raft_consensus.cc:348] T c6b6bd3255d14a0cb254e779b8bde15a P 378cfcc378724919ac8f9f7d19b06bfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "378cfcc378724919ac8f9f7d19b06bfa" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 41349 } } peers { permanent_uuid: "71c25e47472544fa9e042cfcc7bf4f96" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 32929 } } peers { permanent_uuid: "ec8b4c39ed3f419d99b5387ca3e51ca6" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 46215 } }
I20260810 04:07:05.884451 7481 raft_consensus.cc:374] T c6b6bd3255d14a0cb254e779b8bde15a P 378cfcc378724919ac8f9f7d19b06bfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:07:05.884431 7482 consensus_queue.cc:260] T c6b6bd3255d14a0cb254e779b8bde15a P ec8b4c39ed3f419d99b5387ca3e51ca6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "378cfcc378724919ac8f9f7d19b06bfa" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 41349 } } peers { permanent_uuid: "71c25e47472544fa9e042cfcc7bf4f96" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 32929 } } peers { permanent_uuid: "ec8b4c39ed3f419d99b5387ca3e51ca6" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 46215 } }
I20260810 04:07:05.884477 7481 raft_consensus.cc:729] T c6b6bd3255d14a0cb254e779b8bde15a P 378cfcc378724919ac8f9f7d19b06bfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 378cfcc378724919ac8f9f7d19b06bfa, State: Initialized, Role: FOLLOWER
I20260810 04:07:05.884517 7481 consensus_queue.cc:260] T c6b6bd3255d14a0cb254e779b8bde15a P 378cfcc378724919ac8f9f7d19b06bfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "378cfcc378724919ac8f9f7d19b06bfa" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 41349 } } peers { permanent_uuid: "71c25e47472544fa9e042cfcc7bf4f96" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 32929 } } peers { permanent_uuid: "ec8b4c39ed3f419d99b5387ca3e51ca6" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 46215 } }
I20260810 04:07:05.884588 7482 ts_tablet_manager.cc:1428] T c6b6bd3255d14a0cb254e779b8bde15a P ec8b4c39ed3f419d99b5387ca3e51ca6: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260810 04:07:05.884686 7481 ts_tablet_manager.cc:1428] T c6b6bd3255d14a0cb254e779b8bde15a P 378cfcc378724919ac8f9f7d19b06bfa: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260810 04:07:05.885051 7489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:05.884855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:35131 (local address 127.31.32.1:48705)
0810 04:07:05.884981 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0810 04:07:05.885021 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.62:35131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":82,"threads_started":1}
I20260810 04:07:05.885493 7491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:05.885214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:35131 (local address 127.31.32.2:49777)
0810 04:07:05.885391 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0810 04:07:05.885439 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.62:35131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":87,"threads_started":1}
I20260810 04:07:05.885772 7483 tablet_bootstrap.cc:492] T c6b6bd3255d14a0cb254e779b8bde15a P 71c25e47472544fa9e042cfcc7bf4f96: No bootstrap required, opened a new log
I20260810 04:07:05.885824 7483 ts_tablet_manager.cc:1397] T c6b6bd3255d14a0cb254e779b8bde15a P 71c25e47472544fa9e042cfcc7bf4f96: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260810 04:07:05.885964 7483 raft_consensus.cc:348] T c6b6bd3255d14a0cb254e779b8bde15a P 71c25e47472544fa9e042cfcc7bf4f96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "378cfcc378724919ac8f9f7d19b06bfa" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 41349 } } peers { permanent_uuid: "71c25e47472544fa9e042cfcc7bf4f96" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 32929 } } peers { permanent_uuid: "ec8b4c39ed3f419d99b5387ca3e51ca6" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 46215 } }
I20260810 04:07:05.886034 7483 raft_consensus.cc:374] T c6b6bd3255d14a0cb254e779b8bde15a P 71c25e47472544fa9e042cfcc7bf4f96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:07:05.886052 7483 raft_consensus.cc:729] T c6b6bd3255d14a0cb254e779b8bde15a P 71c25e47472544fa9e042cfcc7bf4f96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71c25e47472544fa9e042cfcc7bf4f96, State: Initialized, Role: FOLLOWER
I20260810 04:07:05.886096 7483 consensus_queue.cc:260] T c6b6bd3255d14a0cb254e779b8bde15a P 71c25e47472544fa9e042cfcc7bf4f96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "378cfcc378724919ac8f9f7d19b06bfa" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 41349 } } peers { permanent_uuid: "71c25e47472544fa9e042cfcc7bf4f96" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 32929 } } peers { permanent_uuid: "ec8b4c39ed3f419d99b5387ca3e51ca6" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 46215 } }
I20260810 04:07:05.886241 7483 ts_tablet_manager.cc:1428] T c6b6bd3255d14a0cb254e779b8bde15a P 71c25e47472544fa9e042cfcc7bf4f96: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260810 04:07:05.886480 7237 heartbeater.cc:503] Master 127.31.32.60:33905 was elected leader, sending a full tablet report...
I20260810 04:07:05.886654 7493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:05.886463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:35131 (local address 127.31.32.3:36947)
0810 04:07:05.886581 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0810 04:07:05.886615 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.62:35131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":79,"threads_started":1}
I20260810 04:07:05.886935 7491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:05.886694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:35131 (local address 127.31.32.2:37277)
0810 04:07:05.886875 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0810 04:07:05.886905 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.62:35131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165}
I20260810 04:07:05.886963 7389 heartbeater.cc:503] Master 127.31.32.60:33905 was elected leader, sending a full tablet report...
I20260810 04:07:05.887187 7496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:05.886825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:35131 (local address 127.31.32.3:55397)
0810 04:07:05.887126 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0810 04:07:05.887158 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.62:35131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":71,"threads_started":1}
I20260810 04:07:05.947247 7492 raft_consensus.cc:482] T c6b6bd3255d14a0cb254e779b8bde15a P 71c25e47472544fa9e042cfcc7bf4f96 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 04:07:05.947335 7492 raft_consensus.cc:504] T c6b6bd3255d14a0cb254e779b8bde15a P 71c25e47472544fa9e042cfcc7bf4f96 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "378cfcc378724919ac8f9f7d19b06bfa" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 41349 } } peers { permanent_uuid: "71c25e47472544fa9e042cfcc7bf4f96" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 32929 } } peers { permanent_uuid: "ec8b4c39ed3f419d99b5387ca3e51ca6" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 46215 } }
I20260810 04:07:05.947618 7492 leader_election.cc:288] T c6b6bd3255d14a0cb254e779b8bde15a P 71c25e47472544fa9e042cfcc7bf4f96 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 378cfcc378724919ac8f9f7d19b06bfa (127.31.32.2:41349), ec8b4c39ed3f419d99b5387ca3e51ca6 (127.31.32.1:46215)
I20260810 04:07:05.950754 7479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:05.947852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.3:43079 (local address 127.31.32.1:46215)
0810 04:07:05.947896 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0810 04:07:05.947898 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:07:05.947909 (+ 11us) server_negotiation.cc:407] Connection header received
0810 04:07:05.947964 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:07:05.947965 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:07:05.947977 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:07:05.948040 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:07:05.948372 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:05.948831 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:05.949856 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:05.950412 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:05.950489 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:05.950500 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0810 04:07:05.950649 (+ 149us) server_negotiation.cc:299] Negotiation successful
0810 04:07:05.950686 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260810 04:07:05.950769 7477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:05.947865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.3:57799 (local address 127.31.32.2:41349)
0810 04:07:05.947904 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0810 04:07:05.947905 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 04:07:05.947913 (+ 8us) server_negotiation.cc:407] Connection header received
0810 04:07:05.947964 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:07:05.947965 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:07:05.947977 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:07:05.948040 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:07:05.948369 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:05.948818 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:05.949840 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:05.950419 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:05.950499 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:05.950511 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0810 04:07:05.950671 (+ 160us) server_negotiation.cc:299] Negotiation successful
0810 04:07:05.950706 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260810 04:07:05.950786 7496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:05.947777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.2:41349 (local address 127.31.32.3:57799)
0810 04:07:05.947836 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0810 04:07:05.947847 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 04:07:05.947900 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:07:05.948057 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:07:05.948058 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:07:05.948062 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:07:05.948206 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:05.948208 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:05.948906 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:05.948907 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:05.949730 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:05.949732 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:05.950522 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:05.950523 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:05.950594 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:05.950597 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 04:07:05.950662 (+ 65us) client_negotiation.cc:240] Negotiation successful
0810 04:07:05.950700 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26,"mutex_wait_us":78}
I20260810 04:07:05.950788 7493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:05.947777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.1:46215 (local address 127.31.32.3:43079)
0810 04:07:05.947859 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0810 04:07:05.947869 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 04:07:05.947904 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:07:05.948057 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:07:05.948058 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:07:05.948060 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:07:05.948206 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:05.948208 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:05.948914 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:05.948916 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:05.949737 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:05.949739 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:05.950489 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:05.950490 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:05.950565 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:05.950569 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 04:07:05.950640 (+ 71us) client_negotiation.cc:240] Negotiation successful
0810 04:07:05.950679 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72,"mutex_wait_us":105}
I20260810 04:07:05.951248 7038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6b6bd3255d14a0cb254e779b8bde15a" candidate_uuid: "71c25e47472544fa9e042cfcc7bf4f96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec8b4c39ed3f419d99b5387ca3e51ca6" is_pre_election: true
I20260810 04:07:05.951345 7038 raft_consensus.cc:2454] T c6b6bd3255d14a0cb254e779b8bde15a P ec8b4c39ed3f419d99b5387ca3e51ca6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71c25e47472544fa9e042cfcc7bf4f96 in term 0.
I20260810 04:07:05.951355 7187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6b6bd3255d14a0cb254e779b8bde15a" candidate_uuid: "71c25e47472544fa9e042cfcc7bf4f96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "378cfcc378724919ac8f9f7d19b06bfa" is_pre_election: true
I20260810 04:07:05.951507 7187 raft_consensus.cc:2454] T c6b6bd3255d14a0cb254e779b8bde15a P 378cfcc378724919ac8f9f7d19b06bfa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71c25e47472544fa9e042cfcc7bf4f96 in term 0.
I20260810 04:07:05.951555 7264 leader_election.cc:302] T c6b6bd3255d14a0cb254e779b8bde15a P 71c25e47472544fa9e042cfcc7bf4f96 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71c25e47472544fa9e042cfcc7bf4f96, ec8b4c39ed3f419d99b5387ca3e51ca6; no voters:
I20260810 04:07:05.951653 7492 raft_consensus.cc:2781] T c6b6bd3255d14a0cb254e779b8bde15a P 71c25e47472544fa9e042cfcc7bf4f96 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 04:07:05.951690 7492 raft_consensus.cc:482] T c6b6bd3255d14a0cb254e779b8bde15a P 71c25e47472544fa9e042cfcc7bf4f96 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 04:07:05.951716 7492 raft_consensus.cc:3037] T c6b6bd3255d14a0cb254e779b8bde15a P 71c25e47472544fa9e042cfcc7bf4f96 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:07:05.952476 7492 raft_consensus.cc:504] T c6b6bd3255d14a0cb254e779b8bde15a P 71c25e47472544fa9e042cfcc7bf4f96 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "378cfcc378724919ac8f9f7d19b06bfa" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 41349 } } peers { permanent_uuid: "71c25e47472544fa9e042cfcc7bf4f96" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 32929 } } peers { permanent_uuid: "ec8b4c39ed3f419d99b5387ca3e51ca6" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 46215 } }
I20260810 04:07:05.952611 7492 leader_election.cc:288] T c6b6bd3255d14a0cb254e779b8bde15a P 71c25e47472544fa9e042cfcc7bf4f96 [CANDIDATE]: Term 1 election: Requested vote from peers 378cfcc378724919ac8f9f7d19b06bfa (127.31.32.2:41349), ec8b4c39ed3f419d99b5387ca3e51ca6 (127.31.32.1:46215)
I20260810 04:07:05.952761 7038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6b6bd3255d14a0cb254e779b8bde15a" candidate_uuid: "71c25e47472544fa9e042cfcc7bf4f96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec8b4c39ed3f419d99b5387ca3e51ca6"
I20260810 04:07:05.952775 7187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6b6bd3255d14a0cb254e779b8bde15a" candidate_uuid: "71c25e47472544fa9e042cfcc7bf4f96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "378cfcc378724919ac8f9f7d19b06bfa"
I20260810 04:07:05.952837 7038 raft_consensus.cc:3037] T c6b6bd3255d14a0cb254e779b8bde15a P ec8b4c39ed3f419d99b5387ca3e51ca6 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:07:05.952842 7187 raft_consensus.cc:3037] T c6b6bd3255d14a0cb254e779b8bde15a P 378cfcc378724919ac8f9f7d19b06bfa [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:07:05.953517 7038 raft_consensus.cc:2454] T c6b6bd3255d14a0cb254e779b8bde15a P ec8b4c39ed3f419d99b5387ca3e51ca6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71c25e47472544fa9e042cfcc7bf4f96 in term 1.
I20260810 04:07:05.953586 7187 raft_consensus.cc:2454] T c6b6bd3255d14a0cb254e779b8bde15a P 378cfcc378724919ac8f9f7d19b06bfa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71c25e47472544fa9e042cfcc7bf4f96 in term 1.
I20260810 04:07:05.953717 7264 leader_election.cc:302] T c6b6bd3255d14a0cb254e779b8bde15a P 71c25e47472544fa9e042cfcc7bf4f96 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71c25e47472544fa9e042cfcc7bf4f96, ec8b4c39ed3f419d99b5387ca3e51ca6; no voters:
I20260810 04:07:05.953799 7492 raft_consensus.cc:2781] T c6b6bd3255d14a0cb254e779b8bde15a P 71c25e47472544fa9e042cfcc7bf4f96 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:07:05.953986 7492 raft_consensus.cc:686] T c6b6bd3255d14a0cb254e779b8bde15a P 71c25e47472544fa9e042cfcc7bf4f96 [term 1 LEADER]: Becoming Leader. State: Replica: 71c25e47472544fa9e042cfcc7bf4f96, State: Running, Role: LEADER
I20260810 04:07:05.954079 7492 consensus_queue.cc:237] T c6b6bd3255d14a0cb254e779b8bde15a P 71c25e47472544fa9e042cfcc7bf4f96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "378cfcc378724919ac8f9f7d19b06bfa" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 41349 } } peers { permanent_uuid: "71c25e47472544fa9e042cfcc7bf4f96" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 32929 } } peers { permanent_uuid: "ec8b4c39ed3f419d99b5387ca3e51ca6" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 46215 } }
I20260810 04:07:05.954321 7493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:05.954224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:35131 (local address 127.31.32.3:50413)
0810 04:07:05.954266 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0810 04:07:05.954291 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.62:35131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260810 04:07:05.955075 6888 catalog_manager.cc:5495] T c6b6bd3255d14a0cb254e779b8bde15a P 71c25e47472544fa9e042cfcc7bf4f96 reported cstate change: term changed from 0 to 1, leader changed from <none> to 71c25e47472544fa9e042cfcc7bf4f96 (127.31.32.3). New cstate: current_term: 1 leader_uuid: "71c25e47472544fa9e042cfcc7bf4f96" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "378cfcc378724919ac8f9f7d19b06bfa" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 41349 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "71c25e47472544fa9e042cfcc7bf4f96" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 32929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec8b4c39ed3f419d99b5387ca3e51ca6" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 46215 } health_report { overall_health: UNKNOWN } } }
I20260810 04:07:05.972750 7480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:05.970496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38476 (local address 127.31.32.3:32929)
0810 04:07:05.970569 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0810 04:07:05.970571 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:07:05.970614 (+ 43us) server_negotiation.cc:407] Connection header received
0810 04:07:05.970721 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:07:05.970722 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:07:05.970740 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:07:05.970813 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 04:07:05.970952 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:05.971379 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:05.972078 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:05.972314 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:05.972384 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:05.972508 (+ 124us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 04:07:05.972600 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 04:07:05.972652 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0810 04:07:05.972688 (+ 36us) server_negotiation.cc:299] Negotiation successful
0810 04:07:05.972704 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 04:07:05.973744 7038 raft_consensus.cc:1261] T c6b6bd3255d14a0cb254e779b8bde15a P ec8b4c39ed3f419d99b5387ca3e51ca6 [term 1 FOLLOWER]: Refusing update from remote peer 71c25e47472544fa9e042cfcc7bf4f96: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:07:05.973829 7187 raft_consensus.cc:1261] T c6b6bd3255d14a0cb254e779b8bde15a P 378cfcc378724919ac8f9f7d19b06bfa [term 1 FOLLOWER]: Refusing update from remote peer 71c25e47472544fa9e042cfcc7bf4f96: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:07:05.974009 7492 consensus_queue.cc:1035] T c6b6bd3255d14a0cb254e779b8bde15a P 71c25e47472544fa9e042cfcc7bf4f96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec8b4c39ed3f419d99b5387ca3e51ca6" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 46215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:07:05.974138 7497 consensus_queue.cc:1035] T c6b6bd3255d14a0cb254e779b8bde15a P 71c25e47472544fa9e042cfcc7bf4f96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "378cfcc378724919ac8f9f7d19b06bfa" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 41349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:07:05.974555 7088 heartbeater.cc:503] Master 127.31.32.60:33905 was elected leader, sending a full tablet report...
I20260810 04:07:05.974881 7489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:05.974771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:35131 (local address 127.31.32.1:36703)
0810 04:07:05.974813 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0810 04:07:05.974849 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.62:35131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260810 04:07:05.975365 7491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:05.975184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:35131 (local address 127.31.32.2:43061)
0810 04:07:05.975310 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0810 04:07:05.975337 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.62:35131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
I20260810 04:07:05.976033 7493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:05.975890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:35131 (local address 127.31.32.3:39791)
0810 04:07:05.975962 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0810 04:07:05.975989 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.62:35131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260810 04:07:05.976338 7489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:05.976200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:35131 (local address 127.31.32.1:56151)
0810 04:07:05.976274 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0810 04:07:05.976299 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.62:35131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260810 04:07:05.976532 7493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:05.976376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:35131 (local address 127.31.32.3:40809)
0810 04:07:05.976479 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0810 04:07:05.976507 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.62:35131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260810 04:07:05.977730 7491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:05.977642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:35131 (local address 127.31.32.2:46579)
0810 04:07:05.977674 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0810 04:07:05.977699 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.62:35131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
I20260810 04:07:05.978906 6890 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53374:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 04:07:05.979012 6890 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53374:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 04:07:05.980225 6890 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8b612a0b090e4e71a02370b642ea30ab: Sending DeleteTablet for 3 replicas of tablet c6b6bd3255d14a0cb254e779b8bde15a
I20260810 04:07:05.980567 7315 tablet_service.cc:1514] Processing DeleteTablet for tablet c6b6bd3255d14a0cb254e779b8bde15a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:07:05 UTC) from {username='oryx', principal='oryx/127.31.32.60@KRBTEST.COM'} at 127.0.0.1:38472
I20260810 04:07:05.980578 7164 tablet_service.cc:1514] Processing DeleteTablet for tablet c6b6bd3255d14a0cb254e779b8bde15a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:07:05 UTC) from {username='oryx', principal='oryx/127.31.32.60@KRBTEST.COM'} at 127.0.0.1:54580
I20260810 04:07:05.980691 7016 tablet_service.cc:1514] Processing DeleteTablet for tablet c6b6bd3255d14a0cb254e779b8bde15a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:07:05 UTC) from {username='oryx', principal='oryx/127.31.32.60@KRBTEST.COM'} at 127.0.0.1:35820
I20260810 04:07:05.980888 7516 tablet_replica.cc:330] T c6b6bd3255d14a0cb254e779b8bde15a P ec8b4c39ed3f419d99b5387ca3e51ca6: stopping tablet replica
I20260810 04:07:05.980945 7516 raft_consensus.cc:2229] T c6b6bd3255d14a0cb254e779b8bde15a P ec8b4c39ed3f419d99b5387ca3e51ca6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 04:07:05.981056 31872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskws89Io/build/release/bin/kudu with pid 6846
I20260810 04:07:05.980998 7516 raft_consensus.cc:2258] T c6b6bd3255d14a0cb254e779b8bde15a P ec8b4c39ed3f419d99b5387ca3e51ca6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:07:05.988304 7514 tablet_replica.cc:330] T c6b6bd3255d14a0cb254e779b8bde15a P 71c25e47472544fa9e042cfcc7bf4f96: stopping tablet replica
I20260810 04:07:05.988374 7514 raft_consensus.cc:2229] T c6b6bd3255d14a0cb254e779b8bde15a P 71c25e47472544fa9e042cfcc7bf4f96 [term 1 LEADER]: Raft consensus shutting down.
I20260810 04:07:05.988435 7514 raft_consensus.cc:2258] T c6b6bd3255d14a0cb254e779b8bde15a P 71c25e47472544fa9e042cfcc7bf4f96 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:07:05.988546 7515 tablet_replica.cc:330] T c6b6bd3255d14a0cb254e779b8bde15a P 378cfcc378724919ac8f9f7d19b06bfa: stopping tablet replica
I20260810 04:07:05.988600 7515 raft_consensus.cc:2229] T c6b6bd3255d14a0cb254e779b8bde15a P 378cfcc378724919ac8f9f7d19b06bfa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 04:07:05.988644 7515 raft_consensus.cc:2258] T c6b6bd3255d14a0cb254e779b8bde15a P 378cfcc378724919ac8f9f7d19b06bfa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:07:05.988946 7514 ts_tablet_manager.cc:1905] T c6b6bd3255d14a0cb254e779b8bde15a P 71c25e47472544fa9e042cfcc7bf4f96: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:07:05.990221 7515 ts_tablet_manager.cc:1905] T c6b6bd3255d14a0cb254e779b8bde15a P 378cfcc378724919ac8f9f7d19b06bfa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:07:05.990584 7514 ts_tablet_manager.cc:1918] T c6b6bd3255d14a0cb254e779b8bde15a P 71c25e47472544fa9e042cfcc7bf4f96: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 04:07:05.990696 7514 log.cc:1196] T c6b6bd3255d14a0cb254e779b8bde15a P 71c25e47472544fa9e042cfcc7bf4f96: Deleting WAL directory at /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786334794676589-31872-0/minicluster-data/ts-2/wal/wals/c6b6bd3255d14a0cb254e779b8bde15a
W20260810 04:07:05.991348 7478 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:07:05.991004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.1:46215 (local address 127.0.0.1:35822)
0810 04:07:05.991181 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0810 04:07:05.991192 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 04:07:05.991239 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:07:05.991285 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.1:46215: BlockingWrite error: write error: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":157}
I20260810 04:07:05.991338 31872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskws89Io/build/release/bin/kudu with pid 7097
W20260810 04:07:05.992436 6872 catalog_manager.cc:4584] TS ec8b4c39ed3f419d99b5387ca3e51ca6 (127.31.32.1:46215): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c6b6bd3255d14a0cb254e779b8bde15a: Network error: Client connection negotiation failed: client connection to 127.31.32.1:46215: BlockingWrite error: write error: Connection reset by peer (error 104)
I20260810 04:07:05.992863 7514 ts_tablet_manager.cc:1939] T c6b6bd3255d14a0cb254e779b8bde15a P 71c25e47472544fa9e042cfcc7bf4f96: Deleting consensus metadata
I20260810 04:07:05.995420 6874 catalog_manager.cc:4841] TS 71c25e47472544fa9e042cfcc7bf4f96 (127.31.32.3:32929): tablet c6b6bd3255d14a0cb254e779b8bde15a (table test-table [id=82c417f159f447a08dfbd6f5262e45b7]) successfully deleted
W20260810 04:07:06.000903 7478 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:07:06.000742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.2:41349 (local address 127.0.0.1:54584)
0810 04:07:06.000799 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0810 04:07:06.000809 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 04:07:06.000853 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:07:06.000878 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.2:41349: BlockingWrite error: write error: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":42}
I20260810 04:07:06.001166 31872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskws89Io/build/release/bin/kudu with pid 7245
I20260810 04:07:06.007681 31872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskws89Io/build/release/bin/kudu with pid 6543
I20260810 04:07:06.013962 31872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskws89Io/build/release/bin/kudu with pid 6748
2026-08-10T04:07:06Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (4832 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786334794676589-31872-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 04:07:06 dist-test-slave-8w9v krb5kdc[7520](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 04:07:06 dist-test-slave-8w9v krb5kdc[7520](info): set up 2 sockets
Aug 10 04:07:06 dist-test-slave-8w9v krb5kdc[7520](info): commencing operation
krb5kdc: starting...
W20260810 04:07:07.101500 31872 mini_kdc.cc:121] Time spent starting KDC: real 1.064s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786334794676589-31872-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786334794676589-31872-0/krb5kdc/test-user.keytab.
Aug 10 04:07:07 dist-test-slave-8w9v krb5kdc[7520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334827, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T04:07:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T04:07:07Z Disabled control of system clock
WARNING: no policy specified for oryx/127.31.32.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.32.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.32.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786334794676589-31872-0/krb5kdc/oryx_127.31.32.62.keytab.
Entry for principal oryx/127.31.32.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786334794676589-31872-0/krb5kdc/oryx_127.31.32.62.keytab.
I20260810 04:07:07.218200 31872 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskws89Io/build/release/bin/kudu
/tmp/dist-test-taskws89Io/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786334794676589-31872-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786334794676589-31872-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786334794676589-31872-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786334794676589-31872-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.32.62:39225
--webserver_interface=127.31.32.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786334794676589-31872-0/krb5kdc/oryx_127.31.32.62.keytab
--principal=oryx/127.31.32.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.32.20:36037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.32.62:39225
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786334794676589-31872-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786334794676589-31872-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786334794676589-31872-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:07:07.299343 7535 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:07:07.299580 7535 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:07:07.299620 7535 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:07:07.300778 7535 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:07:07.300848 7535 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260810 04:07:07.300870 7535 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:07:07.300891 7535 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:07:07.300920 7535 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:07:07.300944 7535 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:07:07.302443 7535 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.32.20:36037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786334794676589-31872-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786334794676589-31872-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.32.62:39225
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786334794676589-31872-0/krb5kdc/oryx_127.31.32.62.keytab
--principal=oryx/127.31.32.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.32.62:39225
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786334794676589-31872-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.32.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7535
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786334794676589-31872-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:07:07.302709 7535 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:07:07.302928 7535 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:07:07.307721 7540 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:07:07.307763 7535 server_base.cc:1034] running on GCE node
W20260810 04:07:07.307708 7543 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:07:07.307737 7541 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:07:07.308226 7535 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:07:07.308558 7535 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:07:07.309721 7535 hybrid_clock.cc:648] HybridClock initialized: now 1786334827309716 us; error 36 us; skew 500 ppm
Aug 10 04:07:07 dist-test-slave-8w9v krb5kdc[7520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334827, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.32.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:07:07.312307 7535 init.cc:377] Logged in from keytab as oryx/127.31.32.62@KRBTEST.COM (short username oryx)
I20260810 04:07:07.312721 7535 webserver.cc:460] Webserver started at http://127.31.32.62:42995/ using document root <none> and password file <none>
I20260810 04:07:07.312898 7535 fs_manager.cc:359] Metadata directory not provided
I20260810 04:07:07.312937 7535 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:07:07.313024 7535 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:07:07.313997 7535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786334794676589-31872-0/minicluster-data/master-0/data/instance:
uuid: "2c147e41e15f4984b013b451708c65b3"
format_stamp: "Formatted at 2026-08-10 04:07:07 on dist-test-slave-8w9v"
server_key: "d17965ec959152bb60a66ea1e51347e6"
server_key_iv: "fd315465ce44084ebcb2275689332af9"
server_key_version: "encryptionkey@0"
I20260810 04:07:07.314293 7535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786334794676589-31872-0/minicluster-data/master-0/wal/instance:
uuid: "2c147e41e15f4984b013b451708c65b3"
format_stamp: "Formatted at 2026-08-10 04:07:07 on dist-test-slave-8w9v"
server_key: "d17965ec959152bb60a66ea1e51347e6"
server_key_iv: "fd315465ce44084ebcb2275689332af9"
server_key_version: "encryptionkey@0"
I20260810 04:07:07.315541 7535 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 04:07:07.316258 7551 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:07:07.316423 7535 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 04:07:07.316512 7535 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786334794676589-31872-0/minicluster-data/master-0/data,/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786334794676589-31872-0/minicluster-data/master-0/wal
uuid: "2c147e41e15f4984b013b451708c65b3"
format_stamp: "Formatted at 2026-08-10 04:07:07 on dist-test-slave-8w9v"
server_key: "d17965ec959152bb60a66ea1e51347e6"
server_key_iv: "fd315465ce44084ebcb2275689332af9"
server_key_version: "encryptionkey@0"
I20260810 04:07:07.316565 7535 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786334794676589-31872-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786334794676589-31872-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786334794676589-31872-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:07:07.324904 7535 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:07:07.326584 7535 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:07:07.326689 7535 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:07:07.331123 7535 rpc_server.cc:307] RPC server started. Bound to: 127.31.32.62:39225
I20260810 04:07:07.331243 7616 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.32.62:39225 every 8 connection(s)
I20260810 04:07:07.331508 7535 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786334794676589-31872-0/minicluster-data/master-0/data/info.pb
I20260810 04:07:07.332257 7617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:07:07.332872 31872 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskws89Io/build/release/bin/kudu as pid 7535
I20260810 04:07:07.332971 31872 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786334794676589-31872-0/minicluster-data/master-0/wal/instance
I20260810 04:07:07.333166 31872 external_mini_cluster.cc:1428] Setting key fb534fc6bfbb78914a8c448bcf396dcc
I20260810 04:07:07.335412 7617 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c147e41e15f4984b013b451708c65b3: Bootstrap starting.
I20260810 04:07:07.336052 7617 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2c147e41e15f4984b013b451708c65b3: Neither blocks nor log segments found. Creating new log.
I20260810 04:07:07.336470 7617 log.cc:824] T 00000000000000000000000000000000 P 2c147e41e15f4984b013b451708c65b3: Log is configured to *not* fsync() on all Append() calls
I20260810 04:07:07.337281 7617 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c147e41e15f4984b013b451708c65b3: No bootstrap required, opened a new log
Aug 10 04:07:07 dist-test-slave-8w9v krb5kdc[7520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334827, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.31.32.62@KRBTEST.COM
I20260810 04:07:07.339727 7617 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2c147e41e15f4984b013b451708c65b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c147e41e15f4984b013b451708c65b3" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 39225 } }
I20260810 04:07:07.339888 7617 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2c147e41e15f4984b013b451708c65b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:07:07.339936 7617 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2c147e41e15f4984b013b451708c65b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c147e41e15f4984b013b451708c65b3, State: Initialized, Role: FOLLOWER
I20260810 04:07:07.340056 7617 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2c147e41e15f4984b013b451708c65b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c147e41e15f4984b013b451708c65b3" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 39225 } }
I20260810 04:07:07.340157 7617 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2c147e41e15f4984b013b451708c65b3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 04:07:07.340194 7617 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2c147e41e15f4984b013b451708c65b3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 04:07:07.340238 7617 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2c147e41e15f4984b013b451708c65b3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:07:07.340790 7617 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2c147e41e15f4984b013b451708c65b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c147e41e15f4984b013b451708c65b3" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 39225 } }
I20260810 04:07:07.340909 7617 leader_election.cc:302] T 00000000000000000000000000000000 P 2c147e41e15f4984b013b451708c65b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2c147e41e15f4984b013b451708c65b3; no voters:
I20260810 04:07:07.341081 7617 leader_election.cc:288] T 00000000000000000000000000000000 P 2c147e41e15f4984b013b451708c65b3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 04:07:07.341146 7623 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2c147e41e15f4984b013b451708c65b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:07:07.341336 7617 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2c147e41e15f4984b013b451708c65b3 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:07:07.341552 7623 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2c147e41e15f4984b013b451708c65b3 [term 1 LEADER]: Becoming Leader. State: Replica: 2c147e41e15f4984b013b451708c65b3, State: Running, Role: LEADER
I20260810 04:07:07.341694 7623 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2c147e41e15f4984b013b451708c65b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c147e41e15f4984b013b451708c65b3" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 39225 } }
I20260810 04:07:07.342043 7624 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c147e41e15f4984b013b451708c65b3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2c147e41e15f4984b013b451708c65b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c147e41e15f4984b013b451708c65b3" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 39225 } } }
I20260810 04:07:07.342118 7624 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c147e41e15f4984b013b451708c65b3 [sys.catalog]: This master's current role is: LEADER
I20260810 04:07:07.342250 7624 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c147e41e15f4984b013b451708c65b3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2c147e41e15f4984b013b451708c65b3. Latest consensus state: current_term: 1 leader_uuid: "2c147e41e15f4984b013b451708c65b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c147e41e15f4984b013b451708c65b3" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 39225 } } }
I20260810 04:07:07.342303 7624 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c147e41e15f4984b013b451708c65b3 [sys.catalog]: This master's current role is: LEADER
I20260810 04:07:07.342706 7634 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 04:07:07.343271 7634 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 04:07:07.344624 7621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:07.333685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44924 (local address 127.31.32.62:39225)
0810 04:07:07.333848 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0810 04:07:07.333863 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 04:07:07.333887 (+ 24us) server_negotiation.cc:407] Connection header received
0810 04:07:07.334263 (+ 376us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:07:07.334268 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:07:07.334326 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:07:07.334450 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:07:07.335262 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:07.336237 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:07.337199 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:07.337635 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:07.337717 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:07.340396 (+ 2679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:07:07.340400 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:07:07.340400 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:07:07.340409 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:07:07.342519 (+ 2110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:07:07.343018 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:07:07.343021 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:07:07.343022 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:07:07.343215 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:07:07.343424 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:07:07.343425 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:07:07.343425 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:07:07.343647 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:07:07.343794 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0810 04:07:07.344293 (+ 499us) server_negotiation.cc:299] Negotiation successful
0810 04:07:07.344441 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":90,"threads_started":1}
I20260810 04:07:07.344869 7634 catalog_manager.cc:1347] Generated new cluster ID: 4fe1be82cb804fe388771d38071bab68
I20260810 04:07:07.344916 7634 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 04:07:07.357735 7634 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 04:07:07.358206 7634 catalog_manager.cc:1504] Loading token signing keys...
I20260810 04:07:07.366685 7634 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2c147e41e15f4984b013b451708c65b3: Generated new TSK 0
I20260810 04:07:07.366880 7634 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.31.32.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.32.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.32.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786334794676589-31872-0/krb5kdc/oryx_127.31.32.1.keytab.
Entry for principal oryx/127.31.32.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786334794676589-31872-0/krb5kdc/oryx_127.31.32.1.keytab.
I20260810 04:07:07.399170 31872 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskws89Io/build/release/bin/kudu
/tmp/dist-test-taskws89Io/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786334794676589-31872-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786334794676589-31872-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786334794676589-31872-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786334794676589-31872-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.32.1:0
--local_ip_for_outbound_sockets=127.31.32.1
--webserver_interface=127.31.32.1
--webserver_port=0
--tserver_master_addrs=127.31.32.62:39225
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786334794676589-31872-0/krb5kdc/oryx_127.31.32.1.keytab
--principal=oryx/127.31.32.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.32.20:36037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786334794676589-31872-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786334794676589-31872-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786334794676589-31872-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:07:07.501852 7650 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:07:07.502086 7650 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:07:07.502126 7650 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:07:07.503286 7650 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:07:07.503381 7650 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:07:07.503418 7650 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 04:07:07.503475 7650 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.32.1
I20260810 04:07:07.505089 7650 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.32.20:36037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786334794676589-31872-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786334794676589-31872-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786334794676589-31872-0/krb5kdc/oryx_127.31.32.1.keytab
--principal=oryx/127.31.32.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.32.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786334794676589-31872-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.32.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.31.32.62:39225
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7650
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.32.1
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786334794676589-31872-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:07:07.505347 7650 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:07:07.505564 7650 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:07:07.510456 7661 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:07:07.510468 7657 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:07:07.510529 7650 server_base.cc:1034] running on GCE node
W20260810 04:07:07.510468 7659 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:07:07.510911 7650 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:07:07.511143 7650 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:07:07.512310 7650 hybrid_clock.cc:648] HybridClock initialized: now 1786334827512300 us; error 32 us; skew 500 ppm
Aug 10 04:07:07 dist-test-slave-8w9v krb5kdc[7520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334827, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.32.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:07:07.514773 7650 init.cc:377] Logged in from keytab as oryx/127.31.32.1@KRBTEST.COM (short username oryx)
I20260810 04:07:07.515178 7650 webserver.cc:460] Webserver started at http://127.31.32.1:33765/ using document root <none> and password file <none>
I20260810 04:07:07.515398 7650 fs_manager.cc:359] Metadata directory not provided
I20260810 04:07:07.515465 7650 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:07:07.515619 7650 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:07:07.516683 7650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786334794676589-31872-0/minicluster-data/ts-0/data/instance:
uuid: "1d4dabac174c4e2bbc75a10c2fa03a4c"
format_stamp: "Formatted at 2026-08-10 04:07:07 on dist-test-slave-8w9v"
server_key: "89b8caf75a98b17097db507e07bd086e"
server_key_iv: "3314a20eebe581f7d83cf9d1498025a2"
server_key_version: "encryptionkey@0"
I20260810 04:07:07.517050 7650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786334794676589-31872-0/minicluster-data/ts-0/wal/instance:
uuid: "1d4dabac174c4e2bbc75a10c2fa03a4c"
format_stamp: "Formatted at 2026-08-10 04:07:07 on dist-test-slave-8w9v"
server_key: "89b8caf75a98b17097db507e07bd086e"
server_key_iv: "3314a20eebe581f7d83cf9d1498025a2"
server_key_version: "encryptionkey@0"
I20260810 04:07:07.518354 7650 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 04:07:07.519088 7668 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:07:07.519268 7650 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 04:07:07.519376 7650 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786334794676589-31872-0/minicluster-data/ts-0/data,/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786334794676589-31872-0/minicluster-data/ts-0/wal
uuid: "1d4dabac174c4e2bbc75a10c2fa03a4c"
format_stamp: "Formatted at 2026-08-10 04:07:07 on dist-test-slave-8w9v"
server_key: "89b8caf75a98b17097db507e07bd086e"
server_key_iv: "3314a20eebe581f7d83cf9d1498025a2"
server_key_version: "encryptionkey@0"
I20260810 04:07:07.519456 7650 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786334794676589-31872-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786334794676589-31872-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786334794676589-31872-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:07:07.536499 7650 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:07:07.538304 7650 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:07:07.538470 7650 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:07:07.538972 7650 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:07:07.539047 7650 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:07:07.539080 7650 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:07:07.539125 7650 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:07:07.546320 7650 rpc_server.cc:307] RPC server started. Bound to: 127.31.32.1:46371
Aug 10 04:07:07 dist-test-slave-8w9v krb5kdc[7520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334827, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.32.1@KRBTEST.COM for oryx/127.31.32.62@KRBTEST.COM
I20260810 04:07:07.546401 7792 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.32.1:46371 every 8 connection(s)
I20260810 04:07:07.546700 7650 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786334794676589-31872-0/minicluster-data/ts-0/data/info.pb
I20260810 04:07:07.549717 7621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:07.539485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.1:37619 (local address 127.31.32.62:39225)
0810 04:07:07.539535 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0810 04:07:07.539537 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:07:07.539772 (+ 235us) server_negotiation.cc:407] Connection header received
0810 04:07:07.540369 (+ 597us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:07:07.540372 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:07:07.540386 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:07:07.540448 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:07:07.541479 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:07.542339 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:07.544086 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:07.544428 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:07.544483 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:07.546859 (+ 2376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:07:07.546861 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:07:07.546861 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:07:07.546869 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:07:07.548400 (+ 1531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:07:07.548778 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:07:07.548780 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:07:07.548780 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:07:07.548845 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:07:07.549091 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:07:07.549092 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:07:07.549092 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:07:07.549204 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:07:07.549279 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0810 04:07:07.549547 (+ 268us) server_negotiation.cc:299] Negotiation successful
0810 04:07:07.549627 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260810 04:07:07.549882 7677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:07.539478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:39225 (local address 127.31.32.1:37619)
0810 04:07:07.539629 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0810 04:07:07.539672 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 04:07:07.540231 (+ 559us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:07:07.540564 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:07:07.540569 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:07:07.540976 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:07:07.541301 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:07.541367 (+ 66us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:07.542488 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:07.542490 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:07.543968 (+ 1478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:07.543972 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:07.544516 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:07.544517 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:07.544682 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:07.545231 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:07:07.545240 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:07:07.546771 (+ 1531us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:07:07.548480 (+ 1709us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:07:07.548487 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:07:07.548489 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:07:07.548703 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:07:07.548911 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:07:07.548912 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:07:07.548912 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:07:07.549022 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:07:07.549272 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:07:07.549279 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:07:07.549416 (+ 137us) client_negotiation.cc:769] Sending connection context
0810 04:07:07.549515 (+ 99us) client_negotiation.cc:240] Negotiation successful
0810 04:07:07.549664 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":92,"threads_started":1}
I20260810 04:07:07.550410 7793 heartbeater.cc:347] Connected to a master server at 127.31.32.62:39225
I20260810 04:07:07.550508 7793 heartbeater.cc:464] Registering TS with master...
I20260810 04:07:07.550668 7793 heartbeater.cc:511] Master 127.31.32.62:39225 requested a full tablet report, sending...
I20260810 04:07:07.551198 7568 ts_manager.cc:194] Registered new tserver with Master: 1d4dabac174c4e2bbc75a10c2fa03a4c (127.31.32.1:46371)
I20260810 04:07:07.551800 7568 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.31.32.1@KRBTEST.COM'} at 127.31.32.1:37619
I20260810 04:07:07.554792 31872 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskws89Io/build/release/bin/kudu as pid 7650
I20260810 04:07:07.554917 31872 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786334794676589-31872-0/minicluster-data/ts-0/wal/instance
I20260810 04:07:07.555153 31872 external_mini_cluster.cc:1428] Setting key a392e0dd70b29b5abdf17a542d972244
I20260810 04:07:07.557169 7621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:07.551399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.1:43999 (local address 127.31.32.62:39225)
0810 04:07:07.551435 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0810 04:07:07.551437 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:07:07.551634 (+ 197us) server_negotiation.cc:407] Connection header received
0810 04:07:07.551703 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:07:07.551704 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:07:07.551713 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:07:07.551755 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:07:07.552093 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:07.552558 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:07.553213 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:07.553409 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:07.553481 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:07.554256 (+ 775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:07:07.554258 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:07:07.554258 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:07:07.554266 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:07:07.555695 (+ 1429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:07:07.556139 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:07:07.556140 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:07:07.556140 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:07:07.556306 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:07:07.556555 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:07:07.556556 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:07:07.556556 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:07:07.556768 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:07:07.556831 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0810 04:07:07.557013 (+ 182us) server_negotiation.cc:299] Negotiation successful
0810 04:07:07.557082 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260810 04:07:07.557183 7801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:07.551443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:39225 (local address 127.31.32.1:43999)
0810 04:07:07.551559 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0810 04:07:07.551569 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 04:07:07.551647 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:07:07.551767 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:07:07.551768 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:07:07.551958 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:07:07.552028 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:07.552031 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:07.552647 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:07.552648 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:07.553133 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:07.553134 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:07.553480 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:07.553481 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:07.553544 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:07.553904 (+ 360us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:07:07.553908 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:07:07.554171 (+ 263us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:07:07.555773 (+ 1602us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:07:07.555774 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:07:07.555775 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:07:07.556003 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:07:07.556420 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:07:07.556421 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:07:07.556421 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:07:07.556484 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:07:07.556824 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:07:07.556850 (+ 26us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:07:07.556902 (+ 52us) client_negotiation.cc:769] Sending connection context
0810 04:07:07.556987 (+ 85us) client_negotiation.cc:240] Negotiation successful
0810 04:07:07.557084 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85,"thread_start_us":62,"threads_started":1}
WARNING: no policy specified for oryx/127.31.32.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.32.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.32.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786334794676589-31872-0/krb5kdc/oryx_127.31.32.2.keytab.
Entry for principal oryx/127.31.32.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786334794676589-31872-0/krb5kdc/oryx_127.31.32.2.keytab.
I20260810 04:07:07.580953 31872 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskws89Io/build/release/bin/kudu
/tmp/dist-test-taskws89Io/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786334794676589-31872-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786334794676589-31872-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786334794676589-31872-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786334794676589-31872-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.32.2:0
--local_ip_for_outbound_sockets=127.31.32.2
--webserver_interface=127.31.32.2
--webserver_port=0
--tserver_master_addrs=127.31.32.62:39225
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786334794676589-31872-0/krb5kdc/oryx_127.31.32.2.keytab
--principal=oryx/127.31.32.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.32.20:36037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786334794676589-31872-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786334794676589-31872-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786334794676589-31872-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:07:07.666308 7804 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:07:07.666507 7804 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:07:07.666533 7804 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:07:07.667750 7804 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:07:07.667798 7804 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:07:07.667821 7804 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 04:07:07.667886 7804 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.32.2
I20260810 04:07:07.669473 7804 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.32.20:36037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786334794676589-31872-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786334794676589-31872-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786334794676589-31872-0/krb5kdc/oryx_127.31.32.2.keytab
--principal=oryx/127.31.32.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.32.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786334794676589-31872-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.32.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.31.32.62:39225
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7804
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.32.2
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786334794676589-31872-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:07:07.669735 7804 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:07:07.669987 7804 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:07:07.675405 7812 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:07:07.675452 7810 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:07:07.675487 7809 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:07:07.675661 7804 server_base.cc:1034] running on GCE node
I20260810 04:07:07.675824 7804 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:07:07.676090 7804 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:07:07.677264 7804 hybrid_clock.cc:648] HybridClock initialized: now 1786334827677231 us; error 49 us; skew 500 ppm
Aug 10 04:07:07 dist-test-slave-8w9v krb5kdc[7520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334827, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.32.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:07:07.679723 7804 init.cc:377] Logged in from keytab as oryx/127.31.32.2@KRBTEST.COM (short username oryx)
I20260810 04:07:07.680198 7804 webserver.cc:460] Webserver started at http://127.31.32.2:40093/ using document root <none> and password file <none>
I20260810 04:07:07.680438 7804 fs_manager.cc:359] Metadata directory not provided
I20260810 04:07:07.680505 7804 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:07:07.680651 7804 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:07:07.681684 7804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786334794676589-31872-0/minicluster-data/ts-1/data/instance:
uuid: "f000d641e9094650bbd16cf6b7a35069"
format_stamp: "Formatted at 2026-08-10 04:07:07 on dist-test-slave-8w9v"
server_key: "5cc323dcab1387376f70e6231b53ef48"
server_key_iv: "acd29df5089155f9ab958a5e2de7d558"
server_key_version: "encryptionkey@0"
I20260810 04:07:07.682044 7804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786334794676589-31872-0/minicluster-data/ts-1/wal/instance:
uuid: "f000d641e9094650bbd16cf6b7a35069"
format_stamp: "Formatted at 2026-08-10 04:07:07 on dist-test-slave-8w9v"
server_key: "5cc323dcab1387376f70e6231b53ef48"
server_key_iv: "acd29df5089155f9ab958a5e2de7d558"
server_key_version: "encryptionkey@0"
I20260810 04:07:07.683383 7804 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 04:07:07.684085 7819 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:07:07.684290 7804 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 04:07:07.684397 7804 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786334794676589-31872-0/minicluster-data/ts-1/data,/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786334794676589-31872-0/minicluster-data/ts-1/wal
uuid: "f000d641e9094650bbd16cf6b7a35069"
format_stamp: "Formatted at 2026-08-10 04:07:07 on dist-test-slave-8w9v"
server_key: "5cc323dcab1387376f70e6231b53ef48"
server_key_iv: "acd29df5089155f9ab958a5e2de7d558"
server_key_version: "encryptionkey@0"
I20260810 04:07:07.684485 7804 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786334794676589-31872-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786334794676589-31872-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786334794676589-31872-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:07:07.711426 7804 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:07:07.713267 7804 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:07:07.713430 7804 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:07:07.713945 7804 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:07:07.714016 7804 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:07:07.714059 7804 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:07:07.714087 7804 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 04:07:07 dist-test-slave-8w9v krb5kdc[7520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334827, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.32.2@KRBTEST.COM for oryx/127.31.32.62@KRBTEST.COM
I20260810 04:07:07.721395 7804 rpc_server.cc:307] RPC server started. Bound to: 127.31.32.2:44589
I20260810 04:07:07.721468 7948 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.32.2:44589 every 8 connection(s)
I20260810 04:07:07.721818 7804 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786334794676589-31872-0/minicluster-data/ts-1/data/info.pb
I20260810 04:07:07.724232 7621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:07.714483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.2:47953 (local address 127.31.32.62:39225)
0810 04:07:07.714561 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0810 04:07:07.714564 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:07:07.714715 (+ 151us) server_negotiation.cc:407] Connection header received
0810 04:07:07.715205 (+ 490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:07:07.715207 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:07:07.715220 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:07:07.715303 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:07:07.716224 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:07.717045 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:07.718035 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:07.718254 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:07.718324 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:07.721476 (+ 3152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:07:07.721478 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:07:07.721478 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:07:07.721486 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:07:07.722940 (+ 1454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:07:07.723399 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:07:07.723400 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:07:07.723400 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:07:07.723452 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:07:07.723632 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:07:07.723632 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:07:07.723633 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:07:07.723732 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:07:07.723783 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0810 04:07:07.724030 (+ 247us) server_negotiation.cc:299] Negotiation successful
0810 04:07:07.724091 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260810 04:07:07.724345 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:07.714441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:39225 (local address 127.31.32.2:47953)
0810 04:07:07.714590 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0810 04:07:07.714628 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0810 04:07:07.715091 (+ 463us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:07:07.715398 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:07:07.715402 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:07:07.715801 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:07:07.716111 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:07.716119 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:07.717155 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:07.717156 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:07.717941 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:07.717943 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:07.718390 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:07.718391 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:07.718512 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:07.719058 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:07:07.719068 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:07:07.721372 (+ 2304us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:07:07.723014 (+ 1642us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:07:07.723048 (+ 34us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:07:07.723050 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:07:07.723329 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:07:07.723498 (+ 169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:07:07.723499 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:07:07.723499 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:07:07.723583 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:07:07.723803 (+ 220us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:07:07.723808 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:07:07.723913 (+ 105us) client_negotiation.cc:769] Sending connection context
0810 04:07:07.723994 (+ 81us) client_negotiation.cc:240] Negotiation successful
0810 04:07:07.724107 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":81,"threads_started":1}
I20260810 04:07:07.724788 7949 heartbeater.cc:347] Connected to a master server at 127.31.32.62:39225
I20260810 04:07:07.724871 7949 heartbeater.cc:464] Registering TS with master...
I20260810 04:07:07.725054 7949 heartbeater.cc:511] Master 127.31.32.62:39225 requested a full tablet report, sending...
I20260810 04:07:07.725441 7567 ts_manager.cc:194] Registered new tserver with Master: f000d641e9094650bbd16cf6b7a35069 (127.31.32.2:44589)
I20260810 04:07:07.725970 7567 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.31.32.2@KRBTEST.COM'} at 127.31.32.2:47953
I20260810 04:07:07.726097 31872 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskws89Io/build/release/bin/kudu as pid 7804
I20260810 04:07:07.726222 31872 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786334794676589-31872-0/minicluster-data/ts-1/wal/instance
I20260810 04:07:07.726465 31872 external_mini_cluster.cc:1428] Setting key 76e909f68139ad1d455acc093179c562
I20260810 04:07:07.732326 7621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:07.725826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.2:60689 (local address 127.31.32.62:39225)
0810 04:07:07.725864 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0810 04:07:07.725866 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:07:07.725938 (+ 72us) server_negotiation.cc:407] Connection header received
0810 04:07:07.725987 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:07:07.725988 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:07:07.725999 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:07:07.726062 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:07:07.726526 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:07.727180 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:07.728251 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:07.728430 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:07.728484 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:07.729435 (+ 951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:07:07.729436 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:07:07.729437 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:07:07.729444 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:07:07.731053 (+ 1609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:07:07.731407 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:07:07.731408 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:07:07.731409 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:07:07.731476 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:07:07.731674 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:07:07.731675 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:07:07.731675 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:07:07.731897 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:07:07.731979 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0810 04:07:07.732087 (+ 108us) server_negotiation.cc:299] Negotiation successful
0810 04:07:07.732213 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260810 04:07:07.732564 7955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:07.725728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:39225 (local address 127.31.32.2:60689)
0810 04:07:07.725861 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0810 04:07:07.725872 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 04:07:07.725935 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:07:07.726112 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:07:07.726113 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:07:07.726340 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:07:07.726428 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:07.726430 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:07.727316 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:07.727317 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:07.728171 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:07.728174 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:07.728517 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:07.728518 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:07.728586 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:07.728959 (+ 373us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:07:07.728964 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:07:07.729349 (+ 385us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:07:07.731143 (+ 1794us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:07:07.731143 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:07:07.731144 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:07:07.731310 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:07:07.731554 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:07:07.731555 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:07:07.731555 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:07:07.731607 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:07:07.731971 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:07:07.731971 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:07:07.732005 (+ 34us) client_negotiation.cc:769] Sending connection context
0810 04:07:07.732058 (+ 53us) client_negotiation.cc:240] Negotiation successful
0810 04:07:07.732413 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":79,"threads_started":1}
WARNING: no policy specified for oryx/127.31.32.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.32.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.32.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786334794676589-31872-0/krb5kdc/oryx_127.31.32.3.keytab.
Entry for principal oryx/127.31.32.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786334794676589-31872-0/krb5kdc/oryx_127.31.32.3.keytab.
I20260810 04:07:07.754520 31872 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskws89Io/build/release/bin/kudu
/tmp/dist-test-taskws89Io/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786334794676589-31872-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786334794676589-31872-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786334794676589-31872-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786334794676589-31872-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.32.3:0
--local_ip_for_outbound_sockets=127.31.32.3
--webserver_interface=127.31.32.3
--webserver_port=0
--tserver_master_addrs=127.31.32.62:39225
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786334794676589-31872-0/krb5kdc/oryx_127.31.32.3.keytab
--principal=oryx/127.31.32.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.32.20:36037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786334794676589-31872-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786334794676589-31872-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786334794676589-31872-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:07:07.835639 7958 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:07:07.835880 7958 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:07:07.835923 7958 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:07:07.837330 7958 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:07:07.837395 7958 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:07:07.837432 7958 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 04:07:07.837483 7958 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.32.3
I20260810 04:07:07.838976 7958 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.32.20:36037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786334794676589-31872-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786334794676589-31872-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786334794676589-31872-0/krb5kdc/oryx_127.31.32.3.keytab
--principal=oryx/127.31.32.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.32.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786334794676589-31872-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.32.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.31.32.62:39225
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7958
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.32.3
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786334794676589-31872-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:07:07.839272 7958 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:07:07.839519 7958 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:07:07.844980 7964 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:07:07.844988 7966 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:07:07.845140 7963 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:07:07.845335 7958 server_base.cc:1034] running on GCE node
I20260810 04:07:07.845534 7958 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:07:07.845728 7958 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:07:07.846940 7958 hybrid_clock.cc:648] HybridClock initialized: now 1786334827846926 us; error 36 us; skew 500 ppm
Aug 10 04:07:07 dist-test-slave-8w9v krb5kdc[7520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334827, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.32.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:07:07.849499 7958 init.cc:377] Logged in from keytab as oryx/127.31.32.3@KRBTEST.COM (short username oryx)
I20260810 04:07:07.849962 7958 webserver.cc:460] Webserver started at http://127.31.32.3:44705/ using document root <none> and password file <none>
I20260810 04:07:07.850163 7958 fs_manager.cc:359] Metadata directory not provided
I20260810 04:07:07.850203 7958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:07:07.850299 7958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:07:07.851347 7958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786334794676589-31872-0/minicluster-data/ts-2/data/instance:
uuid: "5e50b5e8a58546bdb6509bba0687350b"
format_stamp: "Formatted at 2026-08-10 04:07:07 on dist-test-slave-8w9v"
server_key: "f8f6d36f0baec73c8decbf69767fc04d"
server_key_iv: "ff7f91f06fdd5982ba8e1418e7243bb7"
server_key_version: "encryptionkey@0"
I20260810 04:07:07.851656 7958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786334794676589-31872-0/minicluster-data/ts-2/wal/instance:
uuid: "5e50b5e8a58546bdb6509bba0687350b"
format_stamp: "Formatted at 2026-08-10 04:07:07 on dist-test-slave-8w9v"
server_key: "f8f6d36f0baec73c8decbf69767fc04d"
server_key_iv: "ff7f91f06fdd5982ba8e1418e7243bb7"
server_key_version: "encryptionkey@0"
I20260810 04:07:07.853055 7958 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 04:07:07.853789 7975 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:07:07.853946 7958 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 04:07:07.854031 7958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786334794676589-31872-0/minicluster-data/ts-2/data,/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786334794676589-31872-0/minicluster-data/ts-2/wal
uuid: "5e50b5e8a58546bdb6509bba0687350b"
format_stamp: "Formatted at 2026-08-10 04:07:07 on dist-test-slave-8w9v"
server_key: "f8f6d36f0baec73c8decbf69767fc04d"
server_key_iv: "ff7f91f06fdd5982ba8e1418e7243bb7"
server_key_version: "encryptionkey@0"
I20260810 04:07:07.854086 7958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786334794676589-31872-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786334794676589-31872-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786334794676589-31872-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:07:07.863286 7958 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:07:07.865545 7958 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:07:07.865650 7958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:07:07.866068 7958 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:07:07.866106 7958 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:07:07.866129 7958 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:07:07.866178 7958 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 04:07:07 dist-test-slave-8w9v krb5kdc[7520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334827, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.32.3@KRBTEST.COM for oryx/127.31.32.62@KRBTEST.COM
I20260810 04:07:07.873291 7958 rpc_server.cc:307] RPC server started. Bound to: 127.31.32.3:36469
I20260810 04:07:07.873401 8099 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.32.3:36469 every 8 connection(s)
I20260810 04:07:07.873688 7958 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786334794676589-31872-0/minicluster-data/ts-2/data/info.pb
I20260810 04:07:07.876514 7621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:07.866536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.3:60843 (local address 127.31.32.62:39225)
0810 04:07:07.866587 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0810 04:07:07.866589 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:07:07.866896 (+ 307us) server_negotiation.cc:407] Connection header received
0810 04:07:07.867553 (+ 657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:07:07.867554 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:07:07.867567 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:07:07.867628 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:07:07.868601 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:07.869338 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:07.870937 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:07.871280 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:07.871350 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:07.873847 (+ 2497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:07:07.873848 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:07:07.873849 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:07:07.873856 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:07:07.875347 (+ 1491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:07:07.875681 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:07:07.875681 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:07:07.875682 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:07:07.875730 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:07:07.875928 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:07:07.875929 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:07:07.875929 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:07:07.876034 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:07:07.876082 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0810 04:07:07.876393 (+ 311us) server_negotiation.cc:299] Negotiation successful
0810 04:07:07.876453 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260810 04:07:07.876683 7984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:07.866543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:39225 (local address 127.31.32.3:60843)
0810 04:07:07.866760 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0810 04:07:07.866800 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0810 04:07:07.867437 (+ 637us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:07:07.867739 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:07:07.867744 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:07:07.868116 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:07:07.868477 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:07.868485 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:07.869412 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:07.869414 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:07.870834 (+ 1420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:07.870838 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:07.871417 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:07.871418 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:07.871523 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:07.872052 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:07:07.872069 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:07:07.873757 (+ 1688us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:07:07.875415 (+ 1658us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:07:07.875418 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:07:07.875420 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:07:07.875617 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:07:07.875786 (+ 169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:07:07.875787 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:07:07.875787 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:07:07.875876 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:07:07.876086 (+ 210us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:07:07.876090 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:07:07.876267 (+ 177us) client_negotiation.cc:769] Sending connection context
0810 04:07:07.876364 (+ 97us) client_negotiation.cc:240] Negotiation successful
0810 04:07:07.876474 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":88,"threads_started":1}
I20260810 04:07:07.877178 8101 heartbeater.cc:347] Connected to a master server at 127.31.32.62:39225
I20260810 04:07:07.877271 8101 heartbeater.cc:464] Registering TS with master...
I20260810 04:07:07.877449 8101 heartbeater.cc:511] Master 127.31.32.62:39225 requested a full tablet report, sending...
I20260810 04:07:07.877799 7567 ts_manager.cc:194] Registered new tserver with Master: 5e50b5e8a58546bdb6509bba0687350b (127.31.32.3:36469)
I20260810 04:07:07.878183 7567 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.31.32.3@KRBTEST.COM'} at 127.31.32.3:60843
I20260810 04:07:07.879393 31872 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskws89Io/build/release/bin/kudu as pid 7958
I20260810 04:07:07.879504 31872 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786334794676589-31872-0/minicluster-data/ts-2/wal/instance
I20260810 04:07:07.879683 31872 external_mini_cluster.cc:1428] Setting key d2dcf9452184ed16a7c695435c55ea67
I20260810 04:07:07.880596 31872 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 04:07:07.883855 7621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:07.878229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.3:44507 (local address 127.31.32.62:39225)
0810 04:07:07.878269 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0810 04:07:07.878270 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 04:07:07.878391 (+ 121us) server_negotiation.cc:407] Connection header received
0810 04:07:07.878442 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:07:07.878442 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:07:07.878452 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:07:07.878507 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:07:07.878812 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:07.879369 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:07.880347 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:07.880619 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:07.880711 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:07.881588 (+ 877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:07:07.881589 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:07:07.881590 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:07:07.881596 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:07:07.882779 (+ 1183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:07:07.883116 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:07:07.883118 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:07:07.883118 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:07:07.883169 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:07:07.883371 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:07:07.883372 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:07:07.883372 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:07:07.883478 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:07:07.883546 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0810 04:07:07.883700 (+ 154us) server_negotiation.cc:299] Negotiation successful
0810 04:07:07.883771 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260810 04:07:07.883883 8107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:07.878176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:39225 (local address 127.31.32.3:44507)
0810 04:07:07.878343 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0810 04:07:07.878352 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 04:07:07.878398 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:07:07.878515 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:07:07.878516 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:07:07.878683 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:07:07.878735 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:07.878737 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:07.879504 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:07.879505 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:07.880225 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:07.880227 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:07.880713 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:07.880714 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:07.880780 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:07.881210 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:07:07.881215 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:07:07.881478 (+ 263us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:07:07.882878 (+ 1400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:07:07.882879 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:07:07.882879 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:07:07.883044 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:07:07.883242 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:07:07.883243 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:07:07.883244 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:07:07.883298 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:07:07.883550 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:07:07.883551 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:07:07.883593 (+ 42us) client_negotiation.cc:769] Sending connection context
0810 04:07:07.883674 (+ 81us) client_negotiation.cc:240] Negotiation successful
0810 04:07:07.883772 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":63,"threads_started":1}
Aug 10 04:07:07 dist-test-slave-8w9v krb5kdc[7520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334827, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 10 04:07:07 dist-test-slave-8w9v krb5kdc[7520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.31.32.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 04:07:07 dist-test-slave-8w9v krb5kdc[7520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.31.32.62@KRBTEST.COM, Server not found in Kerberos database
W20260810 04:07:07.902128 7621 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:07:07.897160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44930 (local address 127.31.32.62:39225)
0810 04:07:07.897223 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0810 04:07:07.897225 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:07:07.897233 (+ 8us) server_negotiation.cc:407] Connection header received
0810 04:07:07.897269 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:07:07.897270 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:07:07.897277 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:07:07.897334 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:07:07.897854 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:07.898286 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:07.898982 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:07.899144 (+ 162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:07.899191 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:07.902021 (+ 2830us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44930: BlockingRecv error: recv got EOF from 127.0.0.1:44930 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Aug 10 04:07:07 dist-test-slave-8w9v krb5kdc[7520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334827, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.31.32.62@KRBTEST.COM
I20260810 04:07:07.909615 7621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:07.902926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44942 (local address 127.31.32.62:39225)
0810 04:07:07.902990 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0810 04:07:07.902993 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:07:07.903094 (+ 101us) server_negotiation.cc:407] Connection header received
0810 04:07:07.903170 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:07:07.903171 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:07:07.903184 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:07:07.903243 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:07:07.903723 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:07.904236 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:07.904905 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:07.905084 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:07.905167 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:07.907182 (+ 2015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:07:07.907183 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:07:07.907183 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:07:07.907190 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:07:07.908510 (+ 1320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:07:07.908838 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:07:07.908839 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:07:07.908840 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:07:07.908894 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:07:07.909093 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:07:07.909094 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:07:07.909094 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:07:07.909197 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:07:07.909294 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0810 04:07:07.909445 (+ 151us) server_negotiation.cc:299] Negotiation successful
0810 04:07:07.909515 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260810 04:07:07.911188 7567 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44942:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 04:07:07.912891 7567 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 04:07:07.919332 8128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:07.915269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.1:46371 (local address 127.0.0.1:35138)
0810 04:07:07.915575 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0810 04:07:07.915585 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 04:07:07.915626 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:07:07.916093 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:07:07.916094 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:07:07.916096 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:07:07.916175 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:07.916176 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:07.917046 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:07.917047 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:07.918139 (+ 1092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:07.918141 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:07.919024 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:07.919025 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:07.919104 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:07.919118 (+ 14us) client_negotiation.cc:769] Sending connection context
0810 04:07:07.919193 (+ 75us) client_negotiation.cc:240] Negotiation successful
0810 04:07:07.919248 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"mutex_wait_us":2,"thread_start_us":87,"threads_started":1}
I20260810 04:07:07.919351 8131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:07.915467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35138 (local address 127.31.32.1:46371)
0810 04:07:07.915914 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0810 04:07:07.915919 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 04:07:07.915928 (+ 9us) server_negotiation.cc:407] Connection header received
0810 04:07:07.915963 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:07:07.915966 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:07:07.915979 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:07:07.916266 (+ 287us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:07:07.916326 (+ 60us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:07.916960 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:07.918248 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:07.918936 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:07.919025 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:07.919155 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0810 04:07:07.919209 (+ 54us) server_negotiation.cc:299] Negotiation successful
0810 04:07:07.919245 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":414,"thread_start_us":85,"threads_started":1}
I20260810 04:07:07.919708 8125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:07.915268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.3:36469 (local address 127.0.0.1:36476)
0810 04:07:07.915484 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0810 04:07:07.915525 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 04:07:07.915599 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:07:07.915989 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:07:07.916009 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:07:07.916012 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:07:07.916079 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:07.916082 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:07.917462 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:07.917463 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:07.918541 (+ 1078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:07.918544 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:07.919461 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:07.919462 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:07.919537 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:07.919541 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 04:07:07.919597 (+ 56us) client_negotiation.cc:240] Negotiation successful
0810 04:07:07.919634 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"mutex_wait_us":22,"thread_start_us":63,"threads_started":1}
I20260810 04:07:07.919788 8129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:07.915381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36476 (local address 127.31.32.3:36469)
0810 04:07:07.915732 (+ 351us) server_negotiation.cc:206] Beginning negotiation
0810 04:07:07.915737 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 04:07:07.915750 (+ 13us) server_negotiation.cc:407] Connection header received
0810 04:07:07.915793 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:07:07.915795 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:07:07.915811 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:07:07.916222 (+ 411us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:07:07.916329 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:07.917380 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:07.918633 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:07.919346 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:07.919448 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:07.919523 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0810 04:07:07.919628 (+ 105us) server_negotiation.cc:299] Negotiation successful
0810 04:07:07.919670 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":88,"threads_started":1}
I20260810 04:07:07.920228 8132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:07.915696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54646 (local address 127.31.32.2:44589)
0810 04:07:07.915891 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0810 04:07:07.915897 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 04:07:07.915909 (+ 12us) server_negotiation.cc:407] Connection header received
0810 04:07:07.915951 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:07:07.915954 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:07:07.915970 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:07:07.916070 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:07:07.916295 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:07.917372 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:07.918578 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:07.919340 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:07.919417 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:07.919705 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0810 04:07:07.920028 (+ 323us) server_negotiation.cc:299] Negotiation successful
0810 04:07:07.920074 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":84,"threads_started":1}
I20260810 04:07:07.920430 8130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:07.915465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.2:44589 (local address 127.0.0.1:54646)
0810 04:07:07.915705 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0810 04:07:07.915712 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0810 04:07:07.915752 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:07:07.916160 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:07:07.916161 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:07:07.916162 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:07:07.916200 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:07.916201 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:07.917472 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:07.917473 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:07.918506 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:07.918509 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:07.919482 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:07.919483 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:07.919550 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:07.919554 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 04:07:07.919769 (+ 215us) client_negotiation.cc:240] Negotiation successful
0810 04:07:07.919819 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":75,"threads_started":1}
I20260810 04:07:07.920605 8028 tablet_service.cc:1467] Processing CreateTablet for tablet 1cf097334b7f4d129b314e73971ec2e6 (DEFAULT_TABLE table=test-table [id=51eee7c7b15e4d38aa1feb9787901dd3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:07:07.920612 7720 tablet_service.cc:1467] Processing CreateTablet for tablet 1cf097334b7f4d129b314e73971ec2e6 (DEFAULT_TABLE table=test-table [id=51eee7c7b15e4d38aa1feb9787901dd3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:07:07.920987 8028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1cf097334b7f4d129b314e73971ec2e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:07:07.920984 7720 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1cf097334b7f4d129b314e73971ec2e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:07:07.921119 7875 tablet_service.cc:1467] Processing CreateTablet for tablet 1cf097334b7f4d129b314e73971ec2e6 (DEFAULT_TABLE table=test-table [id=51eee7c7b15e4d38aa1feb9787901dd3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:07:07.921459 7875 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1cf097334b7f4d129b314e73971ec2e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:07:07.924482 8133 tablet_bootstrap.cc:492] T 1cf097334b7f4d129b314e73971ec2e6 P 5e50b5e8a58546bdb6509bba0687350b: Bootstrap starting.
I20260810 04:07:07.925093 8133 tablet_bootstrap.cc:654] T 1cf097334b7f4d129b314e73971ec2e6 P 5e50b5e8a58546bdb6509bba0687350b: Neither blocks nor log segments found. Creating new log.
I20260810 04:07:07.925392 8133 log.cc:824] T 1cf097334b7f4d129b314e73971ec2e6 P 5e50b5e8a58546bdb6509bba0687350b: Log is configured to *not* fsync() on all Append() calls
I20260810 04:07:07.925624 8135 tablet_bootstrap.cc:492] T 1cf097334b7f4d129b314e73971ec2e6 P f000d641e9094650bbd16cf6b7a35069: Bootstrap starting.
I20260810 04:07:07.926208 8135 tablet_bootstrap.cc:654] T 1cf097334b7f4d129b314e73971ec2e6 P f000d641e9094650bbd16cf6b7a35069: Neither blocks nor log segments found. Creating new log.
I20260810 04:07:07.926373 8133 tablet_bootstrap.cc:492] T 1cf097334b7f4d129b314e73971ec2e6 P 5e50b5e8a58546bdb6509bba0687350b: No bootstrap required, opened a new log
I20260810 04:07:07.926463 8133 ts_tablet_manager.cc:1397] T 1cf097334b7f4d129b314e73971ec2e6 P 5e50b5e8a58546bdb6509bba0687350b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 04:07:07.926543 8135 log.cc:824] T 1cf097334b7f4d129b314e73971ec2e6 P f000d641e9094650bbd16cf6b7a35069: Log is configured to *not* fsync() on all Append() calls
I20260810 04:07:07.926878 8134 tablet_bootstrap.cc:492] T 1cf097334b7f4d129b314e73971ec2e6 P 1d4dabac174c4e2bbc75a10c2fa03a4c: Bootstrap starting.
I20260810 04:07:07.927443 8134 tablet_bootstrap.cc:654] T 1cf097334b7f4d129b314e73971ec2e6 P 1d4dabac174c4e2bbc75a10c2fa03a4c: Neither blocks nor log segments found. Creating new log.
I20260810 04:07:07.927768 8134 log.cc:824] T 1cf097334b7f4d129b314e73971ec2e6 P 1d4dabac174c4e2bbc75a10c2fa03a4c: Log is configured to *not* fsync() on all Append() calls
I20260810 04:07:07.928592 8135 tablet_bootstrap.cc:492] T 1cf097334b7f4d129b314e73971ec2e6 P f000d641e9094650bbd16cf6b7a35069: No bootstrap required, opened a new log
I20260810 04:07:07.928684 8135 ts_tablet_manager.cc:1397] T 1cf097334b7f4d129b314e73971ec2e6 P f000d641e9094650bbd16cf6b7a35069: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260810 04:07:07.928714 8134 tablet_bootstrap.cc:492] T 1cf097334b7f4d129b314e73971ec2e6 P 1d4dabac174c4e2bbc75a10c2fa03a4c: No bootstrap required, opened a new log
I20260810 04:07:07.928774 8134 ts_tablet_manager.cc:1397] T 1cf097334b7f4d129b314e73971ec2e6 P 1d4dabac174c4e2bbc75a10c2fa03a4c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 04:07:07.928743 8133 raft_consensus.cc:348] T 1cf097334b7f4d129b314e73971ec2e6 P 5e50b5e8a58546bdb6509bba0687350b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e50b5e8a58546bdb6509bba0687350b" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 36469 } } peers { permanent_uuid: "1d4dabac174c4e2bbc75a10c2fa03a4c" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 46371 } } peers { permanent_uuid: "f000d641e9094650bbd16cf6b7a35069" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 44589 } }
I20260810 04:07:07.928936 8133 raft_consensus.cc:374] T 1cf097334b7f4d129b314e73971ec2e6 P 5e50b5e8a58546bdb6509bba0687350b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:07:07.929009 8133 raft_consensus.cc:729] T 1cf097334b7f4d129b314e73971ec2e6 P 5e50b5e8a58546bdb6509bba0687350b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e50b5e8a58546bdb6509bba0687350b, State: Initialized, Role: FOLLOWER
I20260810 04:07:07.929132 8133 consensus_queue.cc:260] T 1cf097334b7f4d129b314e73971ec2e6 P 5e50b5e8a58546bdb6509bba0687350b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e50b5e8a58546bdb6509bba0687350b" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 36469 } } peers { permanent_uuid: "1d4dabac174c4e2bbc75a10c2fa03a4c" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 46371 } } peers { permanent_uuid: "f000d641e9094650bbd16cf6b7a35069" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 44589 } }
I20260810 04:07:07.929423 8133 ts_tablet_manager.cc:1428] T 1cf097334b7f4d129b314e73971ec2e6 P 5e50b5e8a58546bdb6509bba0687350b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 04:07:07.929545 8101 heartbeater.cc:503] Master 127.31.32.62:39225 was elected leader, sending a full tablet report...
I20260810 04:07:07.930915 8134 raft_consensus.cc:348] T 1cf097334b7f4d129b314e73971ec2e6 P 1d4dabac174c4e2bbc75a10c2fa03a4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e50b5e8a58546bdb6509bba0687350b" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 36469 } } peers { permanent_uuid: "1d4dabac174c4e2bbc75a10c2fa03a4c" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 46371 } } peers { permanent_uuid: "f000d641e9094650bbd16cf6b7a35069" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 44589 } }
I20260810 04:07:07.931089 8134 raft_consensus.cc:374] T 1cf097334b7f4d129b314e73971ec2e6 P 1d4dabac174c4e2bbc75a10c2fa03a4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:07:07.931012 8135 raft_consensus.cc:348] T 1cf097334b7f4d129b314e73971ec2e6 P f000d641e9094650bbd16cf6b7a35069 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e50b5e8a58546bdb6509bba0687350b" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 36469 } } peers { permanent_uuid: "1d4dabac174c4e2bbc75a10c2fa03a4c" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 46371 } } peers { permanent_uuid: "f000d641e9094650bbd16cf6b7a35069" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 44589 } }
I20260810 04:07:07.931144 8134 raft_consensus.cc:729] T 1cf097334b7f4d129b314e73971ec2e6 P 1d4dabac174c4e2bbc75a10c2fa03a4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d4dabac174c4e2bbc75a10c2fa03a4c, State: Initialized, Role: FOLLOWER
I20260810 04:07:07.931183 8135 raft_consensus.cc:374] T 1cf097334b7f4d129b314e73971ec2e6 P f000d641e9094650bbd16cf6b7a35069 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:07:07.931228 8135 raft_consensus.cc:729] T 1cf097334b7f4d129b314e73971ec2e6 P f000d641e9094650bbd16cf6b7a35069 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f000d641e9094650bbd16cf6b7a35069, State: Initialized, Role: FOLLOWER
I20260810 04:07:07.931248 8134 consensus_queue.cc:260] T 1cf097334b7f4d129b314e73971ec2e6 P 1d4dabac174c4e2bbc75a10c2fa03a4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e50b5e8a58546bdb6509bba0687350b" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 36469 } } peers { permanent_uuid: "1d4dabac174c4e2bbc75a10c2fa03a4c" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 46371 } } peers { permanent_uuid: "f000d641e9094650bbd16cf6b7a35069" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 44589 } }
I20260810 04:07:07.931352 8135 consensus_queue.cc:260] T 1cf097334b7f4d129b314e73971ec2e6 P f000d641e9094650bbd16cf6b7a35069 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e50b5e8a58546bdb6509bba0687350b" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 36469 } } peers { permanent_uuid: "1d4dabac174c4e2bbc75a10c2fa03a4c" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 46371 } } peers { permanent_uuid: "f000d641e9094650bbd16cf6b7a35069" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 44589 } }
I20260810 04:07:07.931529 8134 ts_tablet_manager.cc:1428] T 1cf097334b7f4d129b314e73971ec2e6 P 1d4dabac174c4e2bbc75a10c2fa03a4c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 04:07:07.931597 7793 heartbeater.cc:503] Master 127.31.32.62:39225 was elected leader, sending a full tablet report...
I20260810 04:07:07.931633 8135 ts_tablet_manager.cc:1428] T 1cf097334b7f4d129b314e73971ec2e6 P f000d641e9094650bbd16cf6b7a35069: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 04:07:07.931675 7949 heartbeater.cc:503] Master 127.31.32.62:39225 was elected leader, sending a full tablet report...
W20260810 04:07:07.972452 7950 tablet.cc:2369] T 1cf097334b7f4d129b314e73971ec2e6 P f000d641e9094650bbd16cf6b7a35069: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 04:07:08.047493 7795 tablet.cc:2369] T 1cf097334b7f4d129b314e73971ec2e6 P 1d4dabac174c4e2bbc75a10c2fa03a4c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 04:07:08.124373 8102 tablet.cc:2369] T 1cf097334b7f4d129b314e73971ec2e6 P 5e50b5e8a58546bdb6509bba0687350b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 04:07:08.155213 8142 raft_consensus.cc:482] T 1cf097334b7f4d129b314e73971ec2e6 P f000d641e9094650bbd16cf6b7a35069 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 04:07:08.155404 8142 raft_consensus.cc:504] T 1cf097334b7f4d129b314e73971ec2e6 P f000d641e9094650bbd16cf6b7a35069 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e50b5e8a58546bdb6509bba0687350b" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 36469 } } peers { permanent_uuid: "1d4dabac174c4e2bbc75a10c2fa03a4c" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 46371 } } peers { permanent_uuid: "f000d641e9094650bbd16cf6b7a35069" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 44589 } }
I20260810 04:07:08.155807 8142 leader_election.cc:288] T 1cf097334b7f4d129b314e73971ec2e6 P f000d641e9094650bbd16cf6b7a35069 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e50b5e8a58546bdb6509bba0687350b (127.31.32.3:36469), 1d4dabac174c4e2bbc75a10c2fa03a4c (127.31.32.1:46371)
I20260810 04:07:08.158874 8129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:08.156147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.2:53607 (local address 127.31.32.3:36469)
0810 04:07:08.156197 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0810 04:07:08.156199 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:07:08.156210 (+ 11us) server_negotiation.cc:407] Connection header received
0810 04:07:08.156293 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:07:08.156294 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:07:08.156305 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:07:08.156384 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:07:08.156556 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:08.157127 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:08.158167 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:08.158523 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:08.158600 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:08.158611 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0810 04:07:08.158777 (+ 166us) server_negotiation.cc:299] Negotiation successful
0810 04:07:08.158812 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260810 04:07:08.158989 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:08.156035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.3:36469 (local address 127.31.32.2:53607)
0810 04:07:08.156124 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0810 04:07:08.156161 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0810 04:07:08.156221 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:07:08.156389 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:07:08.156390 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:07:08.156391 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:07:08.156467 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:08.156469 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:08.157204 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:08.157205 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:08.158071 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:08.158074 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:08.158611 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:08.158612 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:08.158695 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:08.158698 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 04:07:08.158755 (+ 57us) client_negotiation.cc:240] Negotiation successful
0810 04:07:08.158797 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260810 04:07:08.159029 8143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:08.156035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.1:46371 (local address 127.31.32.2:57811)
0810 04:07:08.156321 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0810 04:07:08.156337 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 04:07:08.156412 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:07:08.156580 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:07:08.156581 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:07:08.156588 (+ 7us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:07:08.156696 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:08.156698 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:08.157405 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:08.157406 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:08.158196 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:08.158198 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:08.158743 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:08.158744 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:08.158820 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:08.158824 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 04:07:08.158885 (+ 61us) client_negotiation.cc:240] Negotiation successful
0810 04:07:08.158925 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"mutex_wait_us":100,"thread_start_us":135,"threads_started":1}
I20260810 04:07:08.159113 8131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:08.156151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.2:57811 (local address 127.31.32.1:46371)
0810 04:07:08.156204 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0810 04:07:08.156207 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:07:08.156400 (+ 193us) server_negotiation.cc:407] Connection header received
0810 04:07:08.156474 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:07:08.156476 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:07:08.156486 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:07:08.156576 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:07:08.156795 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:08.157301 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:08.158282 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:08.158641 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:08.158737 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:08.158747 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0810 04:07:08.158903 (+ 156us) server_negotiation.cc:299] Negotiation successful
0810 04:07:08.159038 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260810 04:07:08.159426 8051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cf097334b7f4d129b314e73971ec2e6" candidate_uuid: "f000d641e9094650bbd16cf6b7a35069" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e50b5e8a58546bdb6509bba0687350b" is_pre_election: true
I20260810 04:07:08.159490 7741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cf097334b7f4d129b314e73971ec2e6" candidate_uuid: "f000d641e9094650bbd16cf6b7a35069" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d4dabac174c4e2bbc75a10c2fa03a4c" is_pre_election: true
I20260810 04:07:08.159603 8051 raft_consensus.cc:2454] T 1cf097334b7f4d129b314e73971ec2e6 P 5e50b5e8a58546bdb6509bba0687350b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f000d641e9094650bbd16cf6b7a35069 in term 0.
I20260810 04:07:08.159638 7741 raft_consensus.cc:2454] T 1cf097334b7f4d129b314e73971ec2e6 P 1d4dabac174c4e2bbc75a10c2fa03a4c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f000d641e9094650bbd16cf6b7a35069 in term 0.
I20260810 04:07:08.160233 7821 leader_election.cc:302] T 1cf097334b7f4d129b314e73971ec2e6 P f000d641e9094650bbd16cf6b7a35069 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d4dabac174c4e2bbc75a10c2fa03a4c, f000d641e9094650bbd16cf6b7a35069; no voters:
I20260810 04:07:08.160471 8142 raft_consensus.cc:2781] T 1cf097334b7f4d129b314e73971ec2e6 P f000d641e9094650bbd16cf6b7a35069 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 04:07:08.160544 8142 raft_consensus.cc:482] T 1cf097334b7f4d129b314e73971ec2e6 P f000d641e9094650bbd16cf6b7a35069 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 04:07:08.160576 8142 raft_consensus.cc:3037] T 1cf097334b7f4d129b314e73971ec2e6 P f000d641e9094650bbd16cf6b7a35069 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:07:08.161291 8142 raft_consensus.cc:504] T 1cf097334b7f4d129b314e73971ec2e6 P f000d641e9094650bbd16cf6b7a35069 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e50b5e8a58546bdb6509bba0687350b" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 36469 } } peers { permanent_uuid: "1d4dabac174c4e2bbc75a10c2fa03a4c" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 46371 } } peers { permanent_uuid: "f000d641e9094650bbd16cf6b7a35069" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 44589 } }
I20260810 04:07:08.161441 8142 leader_election.cc:288] T 1cf097334b7f4d129b314e73971ec2e6 P f000d641e9094650bbd16cf6b7a35069 [CANDIDATE]: Term 1 election: Requested vote from peers 5e50b5e8a58546bdb6509bba0687350b (127.31.32.3:36469), 1d4dabac174c4e2bbc75a10c2fa03a4c (127.31.32.1:46371)
I20260810 04:07:08.161586 7741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cf097334b7f4d129b314e73971ec2e6" candidate_uuid: "f000d641e9094650bbd16cf6b7a35069" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d4dabac174c4e2bbc75a10c2fa03a4c"
I20260810 04:07:08.161608 8051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cf097334b7f4d129b314e73971ec2e6" candidate_uuid: "f000d641e9094650bbd16cf6b7a35069" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e50b5e8a58546bdb6509bba0687350b"
I20260810 04:07:08.161670 7741 raft_consensus.cc:3037] T 1cf097334b7f4d129b314e73971ec2e6 P 1d4dabac174c4e2bbc75a10c2fa03a4c [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:07:08.161695 8051 raft_consensus.cc:3037] T 1cf097334b7f4d129b314e73971ec2e6 P 5e50b5e8a58546bdb6509bba0687350b [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:07:08.162348 7741 raft_consensus.cc:2454] T 1cf097334b7f4d129b314e73971ec2e6 P 1d4dabac174c4e2bbc75a10c2fa03a4c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f000d641e9094650bbd16cf6b7a35069 in term 1.
I20260810 04:07:08.162485 8051 raft_consensus.cc:2454] T 1cf097334b7f4d129b314e73971ec2e6 P 5e50b5e8a58546bdb6509bba0687350b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f000d641e9094650bbd16cf6b7a35069 in term 1.
I20260810 04:07:08.162523 7821 leader_election.cc:302] T 1cf097334b7f4d129b314e73971ec2e6 P f000d641e9094650bbd16cf6b7a35069 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d4dabac174c4e2bbc75a10c2fa03a4c, f000d641e9094650bbd16cf6b7a35069; no voters:
I20260810 04:07:08.162647 8142 raft_consensus.cc:2781] T 1cf097334b7f4d129b314e73971ec2e6 P f000d641e9094650bbd16cf6b7a35069 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:07:08.162789 8142 raft_consensus.cc:686] T 1cf097334b7f4d129b314e73971ec2e6 P f000d641e9094650bbd16cf6b7a35069 [term 1 LEADER]: Becoming Leader. State: Replica: f000d641e9094650bbd16cf6b7a35069, State: Running, Role: LEADER
I20260810 04:07:08.162987 8142 consensus_queue.cc:237] T 1cf097334b7f4d129b314e73971ec2e6 P f000d641e9094650bbd16cf6b7a35069 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e50b5e8a58546bdb6509bba0687350b" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 36469 } } peers { permanent_uuid: "1d4dabac174c4e2bbc75a10c2fa03a4c" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 46371 } } peers { permanent_uuid: "f000d641e9094650bbd16cf6b7a35069" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 44589 } }
I20260810 04:07:08.163827 7566 catalog_manager.cc:5495] T 1cf097334b7f4d129b314e73971ec2e6 P f000d641e9094650bbd16cf6b7a35069 reported cstate change: term changed from 0 to 1, leader changed from <none> to f000d641e9094650bbd16cf6b7a35069 (127.31.32.2). New cstate: current_term: 1 leader_uuid: "f000d641e9094650bbd16cf6b7a35069" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e50b5e8a58546bdb6509bba0687350b" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 36469 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1d4dabac174c4e2bbc75a10c2fa03a4c" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 46371 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f000d641e9094650bbd16cf6b7a35069" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 44589 } health_report { overall_health: HEALTHY } } }
I20260810 04:07:08.200687 8132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:08.198330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54650 (local address 127.31.32.2:44589)
0810 04:07:08.198408 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0810 04:07:08.198410 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:07:08.198417 (+ 7us) server_negotiation.cc:407] Connection header received
0810 04:07:08.198449 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:07:08.198449 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:07:08.198467 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:07:08.198528 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 04:07:08.198711 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:08.199199 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:08.199934 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:08.200134 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:08.200190 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:08.200345 (+ 155us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 04:07:08.200442 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 04:07:08.200506 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0810 04:07:08.200547 (+ 41us) server_negotiation.cc:299] Negotiation successful
0810 04:07:08.200565 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 04:07:08.202692 7741 raft_consensus.cc:1261] T 1cf097334b7f4d129b314e73971ec2e6 P 1d4dabac174c4e2bbc75a10c2fa03a4c [term 1 FOLLOWER]: Refusing update from remote peer f000d641e9094650bbd16cf6b7a35069: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:07:08.202720 8051 raft_consensus.cc:1261] T 1cf097334b7f4d129b314e73971ec2e6 P 5e50b5e8a58546bdb6509bba0687350b [term 1 FOLLOWER]: Refusing update from remote peer f000d641e9094650bbd16cf6b7a35069: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:07:08.203099 8142 consensus_queue.cc:1035] T 1cf097334b7f4d129b314e73971ec2e6 P f000d641e9094650bbd16cf6b7a35069 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d4dabac174c4e2bbc75a10c2fa03a4c" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 46371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:07:08.203356 8144 consensus_queue.cc:1035] T 1cf097334b7f4d129b314e73971ec2e6 P f000d641e9094650bbd16cf6b7a35069 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e50b5e8a58546bdb6509bba0687350b" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 36469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:07:08.205837 8147 mvcc.cc:205] Tried to move back new op lower bound from 7316827456315023360 to 7316827456156049408. Current Snapshot: MvccSnapshot[applied={T|T < 7316827456315023360}]
I20260810 04:07:08.206239 8148 mvcc.cc:205] Tried to move back new op lower bound from 7316827456315023360 to 7316827456156049408. Current Snapshot: MvccSnapshot[applied={T|T < 7316827456315023360}]
I20260810 04:07:08.208244 7566 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44942:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 04:07:08.208336 7566 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44942:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 04:07:08.209208 7566 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2c147e41e15f4984b013b451708c65b3: Sending DeleteTablet for 3 replicas of tablet 1cf097334b7f4d129b314e73971ec2e6
I20260810 04:07:08.209529 8028 tablet_service.cc:1514] Processing DeleteTablet for tablet 1cf097334b7f4d129b314e73971ec2e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:07:08 UTC) from {username='oryx', principal='oryx/127.31.32.62@KRBTEST.COM'} at 127.0.0.1:36476
I20260810 04:07:08.209589 7720 tablet_service.cc:1514] Processing DeleteTablet for tablet 1cf097334b7f4d129b314e73971ec2e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:07:08 UTC) from {username='oryx', principal='oryx/127.31.32.62@KRBTEST.COM'} at 127.0.0.1:35138
I20260810 04:07:08.209786 8159 tablet_replica.cc:330] T 1cf097334b7f4d129b314e73971ec2e6 P 1d4dabac174c4e2bbc75a10c2fa03a4c: stopping tablet replica
I20260810 04:07:08.209898 8159 raft_consensus.cc:2229] T 1cf097334b7f4d129b314e73971ec2e6 P 1d4dabac174c4e2bbc75a10c2fa03a4c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 04:07:08.210009 8159 raft_consensus.cc:2258] T 1cf097334b7f4d129b314e73971ec2e6 P 1d4dabac174c4e2bbc75a10c2fa03a4c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:07:08.209977 7566 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44942:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 04:07:08.210109 7566 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 04:07:08.210486 8159 ts_tablet_manager.cc:1905] T 1cf097334b7f4d129b314e73971ec2e6 P 1d4dabac174c4e2bbc75a10c2fa03a4c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:07:08.210601 7875 tablet_service.cc:1514] Processing DeleteTablet for tablet 1cf097334b7f4d129b314e73971ec2e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:07:08 UTC) from {username='oryx', principal='oryx/127.31.32.62@KRBTEST.COM'} at 127.0.0.1:54646
I20260810 04:07:08.210834 8161 tablet_replica.cc:330] T 1cf097334b7f4d129b314e73971ec2e6 P f000d641e9094650bbd16cf6b7a35069: stopping tablet replica
I20260810 04:07:08.210933 8161 raft_consensus.cc:2229] T 1cf097334b7f4d129b314e73971ec2e6 P f000d641e9094650bbd16cf6b7a35069 [term 1 LEADER]: Raft consensus shutting down.
I20260810 04:07:08.211021 8161 raft_consensus.cc:2258] T 1cf097334b7f4d129b314e73971ec2e6 P f000d641e9094650bbd16cf6b7a35069 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:07:08.211440 8161 ts_tablet_manager.cc:1905] T 1cf097334b7f4d129b314e73971ec2e6 P f000d641e9094650bbd16cf6b7a35069: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:07:08.211695 7720 tablet_service.cc:1467] Processing CreateTablet for tablet 2f70d875744c4a8ba3f0ccfdff391772 (DEFAULT_TABLE table=test-table [id=90be5f4aa8f44ae7bda5f0464461f01e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:07:08.211724 8028 tablet_service.cc:1467] Processing CreateTablet for tablet 2f70d875744c4a8ba3f0ccfdff391772 (DEFAULT_TABLE table=test-table [id=90be5f4aa8f44ae7bda5f0464461f01e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:07:08.211796 7720 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f70d875744c4a8ba3f0ccfdff391772. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:07:08.211844 8028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f70d875744c4a8ba3f0ccfdff391772. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:07:08.212337 8159 ts_tablet_manager.cc:1918] T 1cf097334b7f4d129b314e73971ec2e6 P 1d4dabac174c4e2bbc75a10c2fa03a4c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 04:07:08.212405 8159 log.cc:1196] T 1cf097334b7f4d129b314e73971ec2e6 P 1d4dabac174c4e2bbc75a10c2fa03a4c: Deleting WAL directory at /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786334794676589-31872-0/minicluster-data/ts-0/wal/wals/1cf097334b7f4d129b314e73971ec2e6
I20260810 04:07:08.212664 8159 ts_tablet_manager.cc:1939] T 1cf097334b7f4d129b314e73971ec2e6 P 1d4dabac174c4e2bbc75a10c2fa03a4c: Deleting consensus metadata
I20260810 04:07:08.213101 8134 tablet_bootstrap.cc:492] T 2f70d875744c4a8ba3f0ccfdff391772 P 1d4dabac174c4e2bbc75a10c2fa03a4c: Bootstrap starting.
I20260810 04:07:08.213096 7553 catalog_manager.cc:4841] TS 1d4dabac174c4e2bbc75a10c2fa03a4c (127.31.32.1:46371): tablet 1cf097334b7f4d129b314e73971ec2e6 (table test-table [id=51eee7c7b15e4d38aa1feb9787901dd3]) successfully deleted
I20260810 04:07:08.213251 8161 ts_tablet_manager.cc:1918] T 1cf097334b7f4d129b314e73971ec2e6 P f000d641e9094650bbd16cf6b7a35069: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 04:07:08.213321 8161 log.cc:1196] T 1cf097334b7f4d129b314e73971ec2e6 P f000d641e9094650bbd16cf6b7a35069: Deleting WAL directory at /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786334794676589-31872-0/minicluster-data/ts-1/wal/wals/1cf097334b7f4d129b314e73971ec2e6
I20260810 04:07:08.213454 8134 tablet_bootstrap.cc:654] T 2f70d875744c4a8ba3f0ccfdff391772 P 1d4dabac174c4e2bbc75a10c2fa03a4c: Neither blocks nor log segments found. Creating new log.
I20260810 04:07:08.213610 8161 ts_tablet_manager.cc:1939] T 1cf097334b7f4d129b314e73971ec2e6 P f000d641e9094650bbd16cf6b7a35069: Deleting consensus metadata
I20260810 04:07:08.213640 8133 tablet_bootstrap.cc:492] T 2f70d875744c4a8ba3f0ccfdff391772 P 5e50b5e8a58546bdb6509bba0687350b: Bootstrap starting.
I20260810 04:07:08.214107 8133 tablet_bootstrap.cc:654] T 2f70d875744c4a8ba3f0ccfdff391772 P 5e50b5e8a58546bdb6509bba0687350b: Neither blocks nor log segments found. Creating new log.
I20260810 04:07:08.214174 7875 tablet_service.cc:1467] Processing CreateTablet for tablet 2f70d875744c4a8ba3f0ccfdff391772 (DEFAULT_TABLE table=test-table [id=90be5f4aa8f44ae7bda5f0464461f01e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:07:08.214280 7875 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f70d875744c4a8ba3f0ccfdff391772. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:07:08.214808 8133 tablet_bootstrap.cc:492] T 2f70d875744c4a8ba3f0ccfdff391772 P 5e50b5e8a58546bdb6509bba0687350b: No bootstrap required, opened a new log
I20260810 04:07:08.214867 8133 ts_tablet_manager.cc:1397] T 2f70d875744c4a8ba3f0ccfdff391772 P 5e50b5e8a58546bdb6509bba0687350b: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260810 04:07:08.215023 8133 raft_consensus.cc:348] T 2f70d875744c4a8ba3f0ccfdff391772 P 5e50b5e8a58546bdb6509bba0687350b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d4dabac174c4e2bbc75a10c2fa03a4c" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 46371 } } peers { permanent_uuid: "f000d641e9094650bbd16cf6b7a35069" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 44589 } } peers { permanent_uuid: "5e50b5e8a58546bdb6509bba0687350b" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 36469 } }
I20260810 04:07:08.215095 8133 raft_consensus.cc:374] T 2f70d875744c4a8ba3f0ccfdff391772 P 5e50b5e8a58546bdb6509bba0687350b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:07:08.215112 8133 raft_consensus.cc:729] T 2f70d875744c4a8ba3f0ccfdff391772 P 5e50b5e8a58546bdb6509bba0687350b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e50b5e8a58546bdb6509bba0687350b, State: Initialized, Role: FOLLOWER
I20260810 04:07:08.215161 8133 consensus_queue.cc:260] T 2f70d875744c4a8ba3f0ccfdff391772 P 5e50b5e8a58546bdb6509bba0687350b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d4dabac174c4e2bbc75a10c2fa03a4c" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 46371 } } peers { permanent_uuid: "f000d641e9094650bbd16cf6b7a35069" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 44589 } } peers { permanent_uuid: "5e50b5e8a58546bdb6509bba0687350b" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 36469 } }
I20260810 04:07:08.215240 8133 ts_tablet_manager.cc:1428] T 2f70d875744c4a8ba3f0ccfdff391772 P 5e50b5e8a58546bdb6509bba0687350b: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260810 04:07:08.215503 8135 tablet_bootstrap.cc:492] T 2f70d875744c4a8ba3f0ccfdff391772 P f000d641e9094650bbd16cf6b7a35069: Bootstrap starting.
I20260810 04:07:08.215567 7553 catalog_manager.cc:4841] TS f000d641e9094650bbd16cf6b7a35069 (127.31.32.2:44589): tablet 1cf097334b7f4d129b314e73971ec2e6 (table test-table [id=51eee7c7b15e4d38aa1feb9787901dd3]) successfully deleted
I20260810 04:07:08.215938 8135 tablet_bootstrap.cc:654] T 2f70d875744c4a8ba3f0ccfdff391772 P f000d641e9094650bbd16cf6b7a35069: Neither blocks nor log segments found. Creating new log.
I20260810 04:07:08.216614 8135 tablet_bootstrap.cc:492] T 2f70d875744c4a8ba3f0ccfdff391772 P f000d641e9094650bbd16cf6b7a35069: No bootstrap required, opened a new log
I20260810 04:07:08.216670 8135 ts_tablet_manager.cc:1397] T 2f70d875744c4a8ba3f0ccfdff391772 P f000d641e9094650bbd16cf6b7a35069: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260810 04:07:08.216816 8134 tablet_bootstrap.cc:492] T 2f70d875744c4a8ba3f0ccfdff391772 P 1d4dabac174c4e2bbc75a10c2fa03a4c: No bootstrap required, opened a new log
I20260810 04:07:08.216866 8134 ts_tablet_manager.cc:1397] T 2f70d875744c4a8ba3f0ccfdff391772 P 1d4dabac174c4e2bbc75a10c2fa03a4c: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260810 04:07:08.216813 8135 raft_consensus.cc:348] T 2f70d875744c4a8ba3f0ccfdff391772 P f000d641e9094650bbd16cf6b7a35069 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d4dabac174c4e2bbc75a10c2fa03a4c" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 46371 } } peers { permanent_uuid: "f000d641e9094650bbd16cf6b7a35069" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 44589 } } peers { permanent_uuid: "5e50b5e8a58546bdb6509bba0687350b" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 36469 } }
I20260810 04:07:08.216912 8135 raft_consensus.cc:374] T 2f70d875744c4a8ba3f0ccfdff391772 P f000d641e9094650bbd16cf6b7a35069 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:07:08.216939 8135 raft_consensus.cc:729] T 2f70d875744c4a8ba3f0ccfdff391772 P f000d641e9094650bbd16cf6b7a35069 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f000d641e9094650bbd16cf6b7a35069, State: Initialized, Role: FOLLOWER
I20260810 04:07:08.216985 8135 consensus_queue.cc:260] T 2f70d875744c4a8ba3f0ccfdff391772 P f000d641e9094650bbd16cf6b7a35069 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d4dabac174c4e2bbc75a10c2fa03a4c" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 46371 } } peers { permanent_uuid: "f000d641e9094650bbd16cf6b7a35069" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 44589 } } peers { permanent_uuid: "5e50b5e8a58546bdb6509bba0687350b" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 36469 } }
I20260810 04:07:08.216991 8134 raft_consensus.cc:348] T 2f70d875744c4a8ba3f0ccfdff391772 P 1d4dabac174c4e2bbc75a10c2fa03a4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d4dabac174c4e2bbc75a10c2fa03a4c" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 46371 } } peers { permanent_uuid: "f000d641e9094650bbd16cf6b7a35069" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 44589 } } peers { permanent_uuid: "5e50b5e8a58546bdb6509bba0687350b" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 36469 } }
I20260810 04:07:08.217051 8134 raft_consensus.cc:374] T 2f70d875744c4a8ba3f0ccfdff391772 P 1d4dabac174c4e2bbc75a10c2fa03a4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:07:08.217072 8134 raft_consensus.cc:729] T 2f70d875744c4a8ba3f0ccfdff391772 P 1d4dabac174c4e2bbc75a10c2fa03a4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d4dabac174c4e2bbc75a10c2fa03a4c, State: Initialized, Role: FOLLOWER
I20260810 04:07:08.217070 8135 ts_tablet_manager.cc:1428] T 2f70d875744c4a8ba3f0ccfdff391772 P f000d641e9094650bbd16cf6b7a35069: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260810 04:07:08.217115 8134 consensus_queue.cc:260] T 2f70d875744c4a8ba3f0ccfdff391772 P 1d4dabac174c4e2bbc75a10c2fa03a4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d4dabac174c4e2bbc75a10c2fa03a4c" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 46371 } } peers { permanent_uuid: "f000d641e9094650bbd16cf6b7a35069" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 44589 } } peers { permanent_uuid: "5e50b5e8a58546bdb6509bba0687350b" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 36469 } }
I20260810 04:07:08.217466 8134 ts_tablet_manager.cc:1428] T 2f70d875744c4a8ba3f0ccfdff391772 P 1d4dabac174c4e2bbc75a10c2fa03a4c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260810 04:07:08.217784 8160 tablet_replica.cc:330] T 1cf097334b7f4d129b314e73971ec2e6 P 5e50b5e8a58546bdb6509bba0687350b: stopping tablet replica
I20260810 04:07:08.218187 8160 raft_consensus.cc:2229] T 1cf097334b7f4d129b314e73971ec2e6 P 5e50b5e8a58546bdb6509bba0687350b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 04:07:08.218304 8160 raft_consensus.cc:2258] T 1cf097334b7f4d129b314e73971ec2e6 P 5e50b5e8a58546bdb6509bba0687350b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:07:08.218706 8160 ts_tablet_manager.cc:1905] T 1cf097334b7f4d129b314e73971ec2e6 P 5e50b5e8a58546bdb6509bba0687350b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:07:08.220155 8160 ts_tablet_manager.cc:1918] T 1cf097334b7f4d129b314e73971ec2e6 P 5e50b5e8a58546bdb6509bba0687350b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 04:07:08.220266 8160 log.cc:1196] T 1cf097334b7f4d129b314e73971ec2e6 P 5e50b5e8a58546bdb6509bba0687350b: Deleting WAL directory at /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786334794676589-31872-0/minicluster-data/ts-2/wal/wals/1cf097334b7f4d129b314e73971ec2e6
I20260810 04:07:08.220582 8160 ts_tablet_manager.cc:1939] T 1cf097334b7f4d129b314e73971ec2e6 P 5e50b5e8a58546bdb6509bba0687350b: Deleting consensus metadata
I20260810 04:07:08.220966 7554 catalog_manager.cc:4841] TS 5e50b5e8a58546bdb6509bba0687350b (127.31.32.3:36469): tablet 1cf097334b7f4d129b314e73971ec2e6 (table test-table [id=51eee7c7b15e4d38aa1feb9787901dd3]) successfully deleted
I20260810 04:07:08.265313 8155 raft_consensus.cc:482] T 2f70d875744c4a8ba3f0ccfdff391772 P f000d641e9094650bbd16cf6b7a35069 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 04:07:08.265425 8155 raft_consensus.cc:504] T 2f70d875744c4a8ba3f0ccfdff391772 P f000d641e9094650bbd16cf6b7a35069 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d4dabac174c4e2bbc75a10c2fa03a4c" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 46371 } } peers { permanent_uuid: "f000d641e9094650bbd16cf6b7a35069" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 44589 } } peers { permanent_uuid: "5e50b5e8a58546bdb6509bba0687350b" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 36469 } }
I20260810 04:07:08.265622 8155 leader_election.cc:288] T 2f70d875744c4a8ba3f0ccfdff391772 P f000d641e9094650bbd16cf6b7a35069 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d4dabac174c4e2bbc75a10c2fa03a4c (127.31.32.1:46371), 5e50b5e8a58546bdb6509bba0687350b (127.31.32.3:36469)
I20260810 04:07:08.265790 7741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f70d875744c4a8ba3f0ccfdff391772" candidate_uuid: "f000d641e9094650bbd16cf6b7a35069" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d4dabac174c4e2bbc75a10c2fa03a4c" is_pre_election: true
I20260810 04:07:08.265914 7741 raft_consensus.cc:2454] T 2f70d875744c4a8ba3f0ccfdff391772 P 1d4dabac174c4e2bbc75a10c2fa03a4c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f000d641e9094650bbd16cf6b7a35069 in term 0.
I20260810 04:07:08.265873 8051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f70d875744c4a8ba3f0ccfdff391772" candidate_uuid: "f000d641e9094650bbd16cf6b7a35069" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e50b5e8a58546bdb6509bba0687350b" is_pre_election: true
I20260810 04:07:08.265957 8051 raft_consensus.cc:2454] T 2f70d875744c4a8ba3f0ccfdff391772 P 5e50b5e8a58546bdb6509bba0687350b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f000d641e9094650bbd16cf6b7a35069 in term 0.
I20260810 04:07:08.266073 7821 leader_election.cc:302] T 2f70d875744c4a8ba3f0ccfdff391772 P f000d641e9094650bbd16cf6b7a35069 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d4dabac174c4e2bbc75a10c2fa03a4c, f000d641e9094650bbd16cf6b7a35069; no voters:
I20260810 04:07:08.266155 8155 raft_consensus.cc:2781] T 2f70d875744c4a8ba3f0ccfdff391772 P f000d641e9094650bbd16cf6b7a35069 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 04:07:08.266184 8155 raft_consensus.cc:482] T 2f70d875744c4a8ba3f0ccfdff391772 P f000d641e9094650bbd16cf6b7a35069 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 04:07:08.266211 8155 raft_consensus.cc:3037] T 2f70d875744c4a8ba3f0ccfdff391772 P f000d641e9094650bbd16cf6b7a35069 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:07:08.266875 8155 raft_consensus.cc:504] T 2f70d875744c4a8ba3f0ccfdff391772 P f000d641e9094650bbd16cf6b7a35069 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d4dabac174c4e2bbc75a10c2fa03a4c" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 46371 } } peers { permanent_uuid: "f000d641e9094650bbd16cf6b7a35069" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 44589 } } peers { permanent_uuid: "5e50b5e8a58546bdb6509bba0687350b" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 36469 } }
I20260810 04:07:08.267055 8155 leader_election.cc:288] T 2f70d875744c4a8ba3f0ccfdff391772 P f000d641e9094650bbd16cf6b7a35069 [CANDIDATE]: Term 1 election: Requested vote from peers 1d4dabac174c4e2bbc75a10c2fa03a4c (127.31.32.1:46371), 5e50b5e8a58546bdb6509bba0687350b (127.31.32.3:36469)
I20260810 04:07:08.267225 8051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f70d875744c4a8ba3f0ccfdff391772" candidate_uuid: "f000d641e9094650bbd16cf6b7a35069" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e50b5e8a58546bdb6509bba0687350b"
I20260810 04:07:08.267211 7741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f70d875744c4a8ba3f0ccfdff391772" candidate_uuid: "f000d641e9094650bbd16cf6b7a35069" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d4dabac174c4e2bbc75a10c2fa03a4c"
I20260810 04:07:08.267289 8051 raft_consensus.cc:3037] T 2f70d875744c4a8ba3f0ccfdff391772 P 5e50b5e8a58546bdb6509bba0687350b [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:07:08.267289 7741 raft_consensus.cc:3037] T 2f70d875744c4a8ba3f0ccfdff391772 P 1d4dabac174c4e2bbc75a10c2fa03a4c [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:07:08.268035 8051 raft_consensus.cc:2454] T 2f70d875744c4a8ba3f0ccfdff391772 P 5e50b5e8a58546bdb6509bba0687350b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f000d641e9094650bbd16cf6b7a35069 in term 1.
I20260810 04:07:08.268035 7741 raft_consensus.cc:2454] T 2f70d875744c4a8ba3f0ccfdff391772 P 1d4dabac174c4e2bbc75a10c2fa03a4c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f000d641e9094650bbd16cf6b7a35069 in term 1.
I20260810 04:07:08.268226 7821 leader_election.cc:302] T 2f70d875744c4a8ba3f0ccfdff391772 P f000d641e9094650bbd16cf6b7a35069 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1d4dabac174c4e2bbc75a10c2fa03a4c, 5e50b5e8a58546bdb6509bba0687350b, f000d641e9094650bbd16cf6b7a35069; no voters:
I20260810 04:07:08.268301 8155 raft_consensus.cc:2781] T 2f70d875744c4a8ba3f0ccfdff391772 P f000d641e9094650bbd16cf6b7a35069 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:07:08.268384 8155 raft_consensus.cc:686] T 2f70d875744c4a8ba3f0ccfdff391772 P f000d641e9094650bbd16cf6b7a35069 [term 1 LEADER]: Becoming Leader. State: Replica: f000d641e9094650bbd16cf6b7a35069, State: Running, Role: LEADER
I20260810 04:07:08.268438 8155 consensus_queue.cc:237] T 2f70d875744c4a8ba3f0ccfdff391772 P f000d641e9094650bbd16cf6b7a35069 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d4dabac174c4e2bbc75a10c2fa03a4c" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 46371 } } peers { permanent_uuid: "f000d641e9094650bbd16cf6b7a35069" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 44589 } } peers { permanent_uuid: "5e50b5e8a58546bdb6509bba0687350b" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 36469 } }
I20260810 04:07:08.269024 7568 catalog_manager.cc:5495] T 2f70d875744c4a8ba3f0ccfdff391772 P f000d641e9094650bbd16cf6b7a35069 reported cstate change: term changed from 0 to 1, leader changed from <none> to f000d641e9094650bbd16cf6b7a35069 (127.31.32.2). New cstate: current_term: 1 leader_uuid: "f000d641e9094650bbd16cf6b7a35069" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d4dabac174c4e2bbc75a10c2fa03a4c" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 46371 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f000d641e9094650bbd16cf6b7a35069" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 44589 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e50b5e8a58546bdb6509bba0687350b" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 36469 } health_report { overall_health: UNKNOWN } } }
Aug 10 04:07:08 dist-test-slave-8w9v krb5kdc[7520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334827, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.32.62@KRBTEST.COM for oryx/127.31.32.62@KRBTEST.COM
I20260810 04:07:08.296523 7621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:08.288883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44950 (local address 127.31.32.62:39225)
0810 04:07:08.288928 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0810 04:07:08.288930 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:07:08.289030 (+ 100us) server_negotiation.cc:407] Connection header received
0810 04:07:08.289098 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:07:08.289098 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:07:08.289106 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:07:08.289153 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:07:08.289767 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:08.290236 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:08.291236 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:08.291400 (+ 164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:08.291449 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:08.293835 (+ 2386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:07:08.293836 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:07:08.293837 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:07:08.293844 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:07:08.295272 (+ 1428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:07:08.295640 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:07:08.295642 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:07:08.295642 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:07:08.295695 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:07:08.295901 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:07:08.295902 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:07:08.295903 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:07:08.296018 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:07:08.296096 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0810 04:07:08.296306 (+ 210us) server_negotiation.cc:299] Negotiation successful
0810 04:07:08.296390 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260810 04:07:08.296595 8171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:08.288770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:39225 (local address 127.0.0.1:44950)
0810 04:07:08.288949 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0810 04:07:08.288961 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 04:07:08.289032 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:07:08.289184 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:07:08.289185 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:07:08.289607 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:07:08.289677 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:08.289680 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:08.290323 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:08.290324 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:08.291133 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:08.291136 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:08.291481 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:08.291483 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:08.291541 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:08.292150 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:07:08.292155 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:07:08.293747 (+ 1592us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:07:08.295368 (+ 1621us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:07:08.295376 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:07:08.295378 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:07:08.295570 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:07:08.295776 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:07:08.295777 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:07:08.295777 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:07:08.295839 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:07:08.296100 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:07:08.296107 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:07:08.296196 (+ 89us) client_negotiation.cc:769] Sending connection context
0810 04:07:08.296284 (+ 88us) client_negotiation.cc:240] Negotiation successful
0810 04:07:08.296460 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"mutex_wait_us":72,"thread_start_us":94,"threads_started":1}
I20260810 04:07:08.298697 7568 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.31.32.62@KRBTEST.COM'} at 127.0.0.1:44950:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260810 04:07:08.298964 7568 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 04:07:08.300490 7720 tablet_service.cc:1467] Processing CreateTablet for tablet e6913f76f6c64a438981736c7074d81a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4f2fe7019dd54c85b2a49bea73d914cf]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 04:07:08.300509 7875 tablet_service.cc:1467] Processing CreateTablet for tablet e6913f76f6c64a438981736c7074d81a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4f2fe7019dd54c85b2a49bea73d914cf]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 04:07:08.300563 8028 tablet_service.cc:1467] Processing CreateTablet for tablet e6913f76f6c64a438981736c7074d81a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4f2fe7019dd54c85b2a49bea73d914cf]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 04:07:08.300648 7875 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6913f76f6c64a438981736c7074d81a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:07:08.300668 7720 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6913f76f6c64a438981736c7074d81a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:07:08.300734 8028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6913f76f6c64a438981736c7074d81a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:07:08.302105 8134 tablet_bootstrap.cc:492] T e6913f76f6c64a438981736c7074d81a P 1d4dabac174c4e2bbc75a10c2fa03a4c: Bootstrap starting.
I20260810 04:07:08.302105 8135 tablet_bootstrap.cc:492] T e6913f76f6c64a438981736c7074d81a P f000d641e9094650bbd16cf6b7a35069: Bootstrap starting.
I20260810 04:07:08.302460 8133 tablet_bootstrap.cc:492] T e6913f76f6c64a438981736c7074d81a P 5e50b5e8a58546bdb6509bba0687350b: Bootstrap starting.
I20260810 04:07:08.302515 8134 tablet_bootstrap.cc:654] T e6913f76f6c64a438981736c7074d81a P 1d4dabac174c4e2bbc75a10c2fa03a4c: Neither blocks nor log segments found. Creating new log.
I20260810 04:07:08.302515 8135 tablet_bootstrap.cc:654] T e6913f76f6c64a438981736c7074d81a P f000d641e9094650bbd16cf6b7a35069: Neither blocks nor log segments found. Creating new log.
I20260810 04:07:08.302906 8133 tablet_bootstrap.cc:654] T e6913f76f6c64a438981736c7074d81a P 5e50b5e8a58546bdb6509bba0687350b: Neither blocks nor log segments found. Creating new log.
I20260810 04:07:08.303303 8134 tablet_bootstrap.cc:492] T e6913f76f6c64a438981736c7074d81a P 1d4dabac174c4e2bbc75a10c2fa03a4c: No bootstrap required, opened a new log
I20260810 04:07:08.303363 8134 ts_tablet_manager.cc:1397] T e6913f76f6c64a438981736c7074d81a P 1d4dabac174c4e2bbc75a10c2fa03a4c: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260810 04:07:08.303511 8134 raft_consensus.cc:348] T e6913f76f6c64a438981736c7074d81a P 1d4dabac174c4e2bbc75a10c2fa03a4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d4dabac174c4e2bbc75a10c2fa03a4c" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 46371 } } peers { permanent_uuid: "f000d641e9094650bbd16cf6b7a35069" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 44589 } } peers { permanent_uuid: "5e50b5e8a58546bdb6509bba0687350b" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 36469 } }
I20260810 04:07:08.303592 8134 raft_consensus.cc:374] T e6913f76f6c64a438981736c7074d81a P 1d4dabac174c4e2bbc75a10c2fa03a4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:07:08.303617 8134 raft_consensus.cc:729] T e6913f76f6c64a438981736c7074d81a P 1d4dabac174c4e2bbc75a10c2fa03a4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d4dabac174c4e2bbc75a10c2fa03a4c, State: Initialized, Role: FOLLOWER
I20260810 04:07:08.303663 8134 consensus_queue.cc:260] T e6913f76f6c64a438981736c7074d81a P 1d4dabac174c4e2bbc75a10c2fa03a4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d4dabac174c4e2bbc75a10c2fa03a4c" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 46371 } } peers { permanent_uuid: "f000d641e9094650bbd16cf6b7a35069" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 44589 } } peers { permanent_uuid: "5e50b5e8a58546bdb6509bba0687350b" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 36469 } }
I20260810 04:07:08.303772 8134 ts_tablet_manager.cc:1428] T e6913f76f6c64a438981736c7074d81a P 1d4dabac174c4e2bbc75a10c2fa03a4c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260810 04:07:08.303884 8133 tablet_bootstrap.cc:492] T e6913f76f6c64a438981736c7074d81a P 5e50b5e8a58546bdb6509bba0687350b: No bootstrap required, opened a new log
I20260810 04:07:08.303927 8133 ts_tablet_manager.cc:1397] T e6913f76f6c64a438981736c7074d81a P 5e50b5e8a58546bdb6509bba0687350b: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260810 04:07:08.303908 8141 tablet_replica.cc:439] T e6913f76f6c64a438981736c7074d81a P 1d4dabac174c4e2bbc75a10c2fa03a4c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d4dabac174c4e2bbc75a10c2fa03a4c" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 46371 } } peers { permanent_uuid: "f000d641e9094650bbd16cf6b7a35069" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 44589 } } peers { permanent_uuid: "5e50b5e8a58546bdb6509bba0687350b" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 36469 } } }
I20260810 04:07:08.303998 8141 tablet_replica.cc:442] T e6913f76f6c64a438981736c7074d81a P 1d4dabac174c4e2bbc75a10c2fa03a4c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 04:07:08.304073 8133 raft_consensus.cc:348] T e6913f76f6c64a438981736c7074d81a P 5e50b5e8a58546bdb6509bba0687350b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d4dabac174c4e2bbc75a10c2fa03a4c" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 46371 } } peers { permanent_uuid: "f000d641e9094650bbd16cf6b7a35069" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 44589 } } peers { permanent_uuid: "5e50b5e8a58546bdb6509bba0687350b" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 36469 } }
I20260810 04:07:08.304155 8133 raft_consensus.cc:374] T e6913f76f6c64a438981736c7074d81a P 5e50b5e8a58546bdb6509bba0687350b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:07:08.304183 8133 raft_consensus.cc:729] T e6913f76f6c64a438981736c7074d81a P 5e50b5e8a58546bdb6509bba0687350b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e50b5e8a58546bdb6509bba0687350b, State: Initialized, Role: FOLLOWER
I20260810 04:07:08.304219 8133 consensus_queue.cc:260] T e6913f76f6c64a438981736c7074d81a P 5e50b5e8a58546bdb6509bba0687350b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d4dabac174c4e2bbc75a10c2fa03a4c" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 46371 } } peers { permanent_uuid: "f000d641e9094650bbd16cf6b7a35069" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 44589 } } peers { permanent_uuid: "5e50b5e8a58546bdb6509bba0687350b" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 36469 } }
I20260810 04:07:08.304315 8133 ts_tablet_manager.cc:1428] T e6913f76f6c64a438981736c7074d81a P 5e50b5e8a58546bdb6509bba0687350b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260810 04:07:08.304421 8139 tablet_replica.cc:439] T e6913f76f6c64a438981736c7074d81a P 5e50b5e8a58546bdb6509bba0687350b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d4dabac174c4e2bbc75a10c2fa03a4c" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 46371 } } peers { permanent_uuid: "f000d641e9094650bbd16cf6b7a35069" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 44589 } } peers { permanent_uuid: "5e50b5e8a58546bdb6509bba0687350b" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 36469 } } }
I20260810 04:07:08.304476 8135 tablet_bootstrap.cc:492] T e6913f76f6c64a438981736c7074d81a P f000d641e9094650bbd16cf6b7a35069: No bootstrap required, opened a new log
I20260810 04:07:08.304494 8139 tablet_replica.cc:442] T e6913f76f6c64a438981736c7074d81a P 5e50b5e8a58546bdb6509bba0687350b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 04:07:08.304530 8135 ts_tablet_manager.cc:1397] T e6913f76f6c64a438981736c7074d81a P f000d641e9094650bbd16cf6b7a35069: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 04:07:08.304847 8135 raft_consensus.cc:348] T e6913f76f6c64a438981736c7074d81a P f000d641e9094650bbd16cf6b7a35069 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d4dabac174c4e2bbc75a10c2fa03a4c" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 46371 } } peers { permanent_uuid: "f000d641e9094650bbd16cf6b7a35069" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 44589 } } peers { permanent_uuid: "5e50b5e8a58546bdb6509bba0687350b" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 36469 } }
I20260810 04:07:08.304950 8135 raft_consensus.cc:374] T e6913f76f6c64a438981736c7074d81a P f000d641e9094650bbd16cf6b7a35069 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:07:08.304975 8135 raft_consensus.cc:729] T e6913f76f6c64a438981736c7074d81a P f000d641e9094650bbd16cf6b7a35069 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f000d641e9094650bbd16cf6b7a35069, State: Initialized, Role: FOLLOWER
I20260810 04:07:08.305022 8135 consensus_queue.cc:260] T e6913f76f6c64a438981736c7074d81a P f000d641e9094650bbd16cf6b7a35069 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d4dabac174c4e2bbc75a10c2fa03a4c" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 46371 } } peers { permanent_uuid: "f000d641e9094650bbd16cf6b7a35069" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 44589 } } peers { permanent_uuid: "5e50b5e8a58546bdb6509bba0687350b" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 36469 } }
I20260810 04:07:08.305114 8135 ts_tablet_manager.cc:1428] T e6913f76f6c64a438981736c7074d81a P f000d641e9094650bbd16cf6b7a35069: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260810 04:07:08.305146 8155 tablet_replica.cc:439] T e6913f76f6c64a438981736c7074d81a P f000d641e9094650bbd16cf6b7a35069: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d4dabac174c4e2bbc75a10c2fa03a4c" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 46371 } } peers { permanent_uuid: "f000d641e9094650bbd16cf6b7a35069" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 44589 } } peers { permanent_uuid: "5e50b5e8a58546bdb6509bba0687350b" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 36469 } } }
I20260810 04:07:08.305236 8155 tablet_replica.cc:442] T e6913f76f6c64a438981736c7074d81a P f000d641e9094650bbd16cf6b7a35069: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 04:07:08.367743 8139 raft_consensus.cc:482] T e6913f76f6c64a438981736c7074d81a P 5e50b5e8a58546bdb6509bba0687350b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 04:07:08.367847 8139 raft_consensus.cc:504] T e6913f76f6c64a438981736c7074d81a P 5e50b5e8a58546bdb6509bba0687350b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d4dabac174c4e2bbc75a10c2fa03a4c" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 46371 } } peers { permanent_uuid: "f000d641e9094650bbd16cf6b7a35069" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 44589 } } peers { permanent_uuid: "5e50b5e8a58546bdb6509bba0687350b" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 36469 } }
I20260810 04:07:08.368247 8139 leader_election.cc:288] T e6913f76f6c64a438981736c7074d81a P 5e50b5e8a58546bdb6509bba0687350b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d4dabac174c4e2bbc75a10c2fa03a4c (127.31.32.1:46371), f000d641e9094650bbd16cf6b7a35069 (127.31.32.2:44589)
I20260810 04:07:08.371109 7984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:08.368462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.1:46371 (local address 127.31.32.3:38611)
0810 04:07:08.368531 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0810 04:07:08.368545 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 04:07:08.368600 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:07:08.368783 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:07:08.368784 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:07:08.368786 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:07:08.368864 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:08.368866 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:08.369514 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:08.369515 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:08.370279 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:08.370281 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:08.370838 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:08.370839 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:08.370913 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:08.370916 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 04:07:08.370981 (+ 65us) client_negotiation.cc:240] Negotiation successful
0810 04:07:08.371019 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260810 04:07:08.371109 8131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:08.368553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.3:38611 (local address 127.31.32.1:46371)
0810 04:07:08.368626 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0810 04:07:08.368628 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:07:08.368654 (+ 26us) server_negotiation.cc:407] Connection header received
0810 04:07:08.368685 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:07:08.368686 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:07:08.368696 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:07:08.368752 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:07:08.368926 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:08.369427 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:08.370373 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:08.370745 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:08.370837 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:08.370849 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0810 04:07:08.370987 (+ 138us) server_negotiation.cc:299] Negotiation successful
0810 04:07:08.371025 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260810 04:07:08.371376 7741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6913f76f6c64a438981736c7074d81a" candidate_uuid: "5e50b5e8a58546bdb6509bba0687350b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d4dabac174c4e2bbc75a10c2fa03a4c" is_pre_election: true
I20260810 04:07:08.371482 7741 raft_consensus.cc:2454] T e6913f76f6c64a438981736c7074d81a P 1d4dabac174c4e2bbc75a10c2fa03a4c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e50b5e8a58546bdb6509bba0687350b in term 0.
I20260810 04:07:08.371644 7977 leader_election.cc:302] T e6913f76f6c64a438981736c7074d81a P 5e50b5e8a58546bdb6509bba0687350b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d4dabac174c4e2bbc75a10c2fa03a4c, 5e50b5e8a58546bdb6509bba0687350b; no voters:
I20260810 04:07:08.371817 8139 raft_consensus.cc:2781] T e6913f76f6c64a438981736c7074d81a P 5e50b5e8a58546bdb6509bba0687350b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 04:07:08.371877 8139 raft_consensus.cc:482] T e6913f76f6c64a438981736c7074d81a P 5e50b5e8a58546bdb6509bba0687350b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 04:07:08.371909 8139 raft_consensus.cc:3037] T e6913f76f6c64a438981736c7074d81a P 5e50b5e8a58546bdb6509bba0687350b [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:07:08.372225 8132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:08.368765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.3:36237 (local address 127.31.32.2:44589)
0810 04:07:08.368838 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0810 04:07:08.368840 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:07:08.368846 (+ 6us) server_negotiation.cc:407] Connection header received
0810 04:07:08.368872 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:07:08.368872 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:07:08.368878 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:07:08.368942 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:07:08.369149 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:08.369626 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:08.371342 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:08.371865 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:08.371931 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:08.371942 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0810 04:07:08.372105 (+ 163us) server_negotiation.cc:299] Negotiation successful
0810 04:07:08.372164 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260810 04:07:08.372233 8175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:08.368604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.2:44589 (local address 127.31.32.3:36237)
0810 04:07:08.368750 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0810 04:07:08.368760 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 04:07:08.368821 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:07:08.368950 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:07:08.368951 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:07:08.368953 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:07:08.369067 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:08.369069 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:08.369705 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:08.369706 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:08.371205 (+ 1499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:08.371209 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:08.371938 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:08.371939 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:08.372017 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:08.372021 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 04:07:08.372095 (+ 74us) client_negotiation.cc:240] Negotiation successful
0810 04:07:08.372154 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":75,"threads_started":1}
I20260810 04:07:08.372469 7899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6913f76f6c64a438981736c7074d81a" candidate_uuid: "5e50b5e8a58546bdb6509bba0687350b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f000d641e9094650bbd16cf6b7a35069" is_pre_election: true
I20260810 04:07:08.372604 7899 raft_consensus.cc:2454] T e6913f76f6c64a438981736c7074d81a P f000d641e9094650bbd16cf6b7a35069 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e50b5e8a58546bdb6509bba0687350b in term 0.
I20260810 04:07:08.372892 8139 raft_consensus.cc:504] T e6913f76f6c64a438981736c7074d81a P 5e50b5e8a58546bdb6509bba0687350b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d4dabac174c4e2bbc75a10c2fa03a4c" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 46371 } } peers { permanent_uuid: "f000d641e9094650bbd16cf6b7a35069" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 44589 } } peers { permanent_uuid: "5e50b5e8a58546bdb6509bba0687350b" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 36469 } }
I20260810 04:07:08.373030 8139 leader_election.cc:288] T e6913f76f6c64a438981736c7074d81a P 5e50b5e8a58546bdb6509bba0687350b [CANDIDATE]: Term 1 election: Requested vote from peers 1d4dabac174c4e2bbc75a10c2fa03a4c (127.31.32.1:46371), f000d641e9094650bbd16cf6b7a35069 (127.31.32.2:44589)
I20260810 04:07:08.373224 7741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6913f76f6c64a438981736c7074d81a" candidate_uuid: "5e50b5e8a58546bdb6509bba0687350b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d4dabac174c4e2bbc75a10c2fa03a4c"
I20260810 04:07:08.373296 7741 raft_consensus.cc:3037] T e6913f76f6c64a438981736c7074d81a P 1d4dabac174c4e2bbc75a10c2fa03a4c [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:07:08.373971 7741 raft_consensus.cc:2454] T e6913f76f6c64a438981736c7074d81a P 1d4dabac174c4e2bbc75a10c2fa03a4c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e50b5e8a58546bdb6509bba0687350b in term 1.
I20260810 04:07:08.374084 7899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6913f76f6c64a438981736c7074d81a" candidate_uuid: "5e50b5e8a58546bdb6509bba0687350b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f000d641e9094650bbd16cf6b7a35069"
I20260810 04:07:08.374140 7977 leader_election.cc:302] T e6913f76f6c64a438981736c7074d81a P 5e50b5e8a58546bdb6509bba0687350b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d4dabac174c4e2bbc75a10c2fa03a4c, 5e50b5e8a58546bdb6509bba0687350b; no voters:
I20260810 04:07:08.374163 7899 raft_consensus.cc:3037] T e6913f76f6c64a438981736c7074d81a P f000d641e9094650bbd16cf6b7a35069 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:07:08.374214 8139 raft_consensus.cc:2781] T e6913f76f6c64a438981736c7074d81a P 5e50b5e8a58546bdb6509bba0687350b [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:07:08.374346 8139 raft_consensus.cc:686] T e6913f76f6c64a438981736c7074d81a P 5e50b5e8a58546bdb6509bba0687350b [term 1 LEADER]: Becoming Leader. State: Replica: 5e50b5e8a58546bdb6509bba0687350b, State: Running, Role: LEADER
I20260810 04:07:08.374415 8139 consensus_queue.cc:237] T e6913f76f6c64a438981736c7074d81a P 5e50b5e8a58546bdb6509bba0687350b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d4dabac174c4e2bbc75a10c2fa03a4c" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 46371 } } peers { permanent_uuid: "f000d641e9094650bbd16cf6b7a35069" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 44589 } } peers { permanent_uuid: "5e50b5e8a58546bdb6509bba0687350b" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 36469 } }
I20260810 04:07:08.374979 8176 tablet_replica.cc:439] T e6913f76f6c64a438981736c7074d81a P 5e50b5e8a58546bdb6509bba0687350b: TxnStatusTablet state changed. Reason: New leader 5e50b5e8a58546bdb6509bba0687350b. Latest consensus state: current_term: 1 leader_uuid: "5e50b5e8a58546bdb6509bba0687350b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d4dabac174c4e2bbc75a10c2fa03a4c" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 46371 } } peers { permanent_uuid: "f000d641e9094650bbd16cf6b7a35069" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 44589 } } peers { permanent_uuid: "5e50b5e8a58546bdb6509bba0687350b" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 36469 } } }
I20260810 04:07:08.375058 7899 raft_consensus.cc:2454] T e6913f76f6c64a438981736c7074d81a P f000d641e9094650bbd16cf6b7a35069 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e50b5e8a58546bdb6509bba0687350b in term 1.
I20260810 04:07:08.375072 8176 tablet_replica.cc:442] T e6913f76f6c64a438981736c7074d81a P 5e50b5e8a58546bdb6509bba0687350b: This TxnStatusTablet replica's current role is: LEADER
I20260810 04:07:08.375202 7568 catalog_manager.cc:5495] T e6913f76f6c64a438981736c7074d81a P 5e50b5e8a58546bdb6509bba0687350b reported cstate change: term changed from 0 to 1, leader changed from <none> to 5e50b5e8a58546bdb6509bba0687350b (127.31.32.3). New cstate: current_term: 1 leader_uuid: "5e50b5e8a58546bdb6509bba0687350b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d4dabac174c4e2bbc75a10c2fa03a4c" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 46371 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f000d641e9094650bbd16cf6b7a35069" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 44589 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e50b5e8a58546bdb6509bba0687350b" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 36469 } health_report { overall_health: HEALTHY } } }
I20260810 04:07:08.375288 8178 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260810 04:07:08.375414 8178 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260810 04:07:08.381320 8129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:08.378967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36480 (local address 127.31.32.3:36469)
0810 04:07:08.379010 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0810 04:07:08.379012 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:07:08.379021 (+ 9us) server_negotiation.cc:407] Connection header received
0810 04:07:08.379095 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:07:08.379095 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:07:08.379121 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:07:08.379179 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 04:07:08.379316 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:08.379743 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:08.380537 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:08.380731 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:08.380825 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:08.381012 (+ 187us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 04:07:08.381108 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 04:07:08.381188 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0810 04:07:08.381243 (+ 55us) server_negotiation.cc:299] Negotiation successful
0810 04:07:08.381262 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260810 04:07:08.381358 8171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:08.378893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.3:36469 (local address 127.0.0.1:36480)
0810 04:07:08.378964 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0810 04:07:08.378974 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 04:07:08.379020 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:07:08.379182 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:07:08.379183 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:07:08.379184 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 04:07:08.379225 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:08.379244 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:08.379824 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:08.379825 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:08.380445 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:08.380447 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:08.380815 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:08.380817 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:08.380889 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:08.380912 (+ 23us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 04:07:08.381175 (+ 263us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 04:07:08.381185 (+ 10us) client_negotiation.cc:769] Sending connection context
0810 04:07:08.381238 (+ 53us) client_negotiation.cc:240] Negotiation successful
0810 04:07:08.381283 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260810 04:07:08.382339 7741 raft_consensus.cc:1261] T e6913f76f6c64a438981736c7074d81a P 1d4dabac174c4e2bbc75a10c2fa03a4c [term 1 FOLLOWER]: Refusing update from remote peer 5e50b5e8a58546bdb6509bba0687350b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:07:08.382570 7899 raft_consensus.cc:1261] T e6913f76f6c64a438981736c7074d81a P f000d641e9094650bbd16cf6b7a35069 [term 1 FOLLOWER]: Refusing update from remote peer 5e50b5e8a58546bdb6509bba0687350b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:07:08.382589 8176 consensus_queue.cc:1035] T e6913f76f6c64a438981736c7074d81a P 5e50b5e8a58546bdb6509bba0687350b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d4dabac174c4e2bbc75a10c2fa03a4c" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 46371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:07:08.382843 8176 consensus_queue.cc:1035] T e6913f76f6c64a438981736c7074d81a P 5e50b5e8a58546bdb6509bba0687350b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f000d641e9094650bbd16cf6b7a35069" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 44589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:07:08.383198 8141 tablet_replica.cc:439] T e6913f76f6c64a438981736c7074d81a P 1d4dabac174c4e2bbc75a10c2fa03a4c: TxnStatusTablet state changed. Reason: New leader 5e50b5e8a58546bdb6509bba0687350b. Latest consensus state: current_term: 1 leader_uuid: "5e50b5e8a58546bdb6509bba0687350b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d4dabac174c4e2bbc75a10c2fa03a4c" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 46371 } } peers { permanent_uuid: "f000d641e9094650bbd16cf6b7a35069" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 44589 } } peers { permanent_uuid: "5e50b5e8a58546bdb6509bba0687350b" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 36469 } } }
I20260810 04:07:08.383282 8141 tablet_replica.cc:442] T e6913f76f6c64a438981736c7074d81a P 1d4dabac174c4e2bbc75a10c2fa03a4c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 04:07:08.383679 8155 tablet_replica.cc:439] T e6913f76f6c64a438981736c7074d81a P f000d641e9094650bbd16cf6b7a35069: TxnStatusTablet state changed. Reason: New leader 5e50b5e8a58546bdb6509bba0687350b. Latest consensus state: current_term: 1 leader_uuid: "5e50b5e8a58546bdb6509bba0687350b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d4dabac174c4e2bbc75a10c2fa03a4c" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 46371 } } peers { permanent_uuid: "f000d641e9094650bbd16cf6b7a35069" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 44589 } } peers { permanent_uuid: "5e50b5e8a58546bdb6509bba0687350b" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 36469 } } }
I20260810 04:07:08.383760 8155 tablet_replica.cc:442] T e6913f76f6c64a438981736c7074d81a P f000d641e9094650bbd16cf6b7a35069: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 04:07:08.384191 8149 mvcc.cc:205] Tried to move back new op lower bound from 7316827457052446720 to 7316827457022226432. Current Snapshot: MvccSnapshot[applied={T|T < 7316827457052446720}]
I20260810 04:07:08.384264 8139 tablet_replica.cc:439] T e6913f76f6c64a438981736c7074d81a P 5e50b5e8a58546bdb6509bba0687350b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5e50b5e8a58546bdb6509bba0687350b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d4dabac174c4e2bbc75a10c2fa03a4c" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 46371 } } peers { permanent_uuid: "f000d641e9094650bbd16cf6b7a35069" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 44589 } } peers { permanent_uuid: "5e50b5e8a58546bdb6509bba0687350b" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 36469 } } }
I20260810 04:07:08.384335 8139 tablet_replica.cc:442] T e6913f76f6c64a438981736c7074d81a P 5e50b5e8a58546bdb6509bba0687350b: This TxnStatusTablet replica's current role is: LEADER
I20260810 04:07:08.384476 8141 tablet_replica.cc:439] T e6913f76f6c64a438981736c7074d81a P 1d4dabac174c4e2bbc75a10c2fa03a4c: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5e50b5e8a58546bdb6509bba0687350b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d4dabac174c4e2bbc75a10c2fa03a4c" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 46371 } } peers { permanent_uuid: "f000d641e9094650bbd16cf6b7a35069" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 44589 } } peers { permanent_uuid: "5e50b5e8a58546bdb6509bba0687350b" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 36469 } } }
I20260810 04:07:08.384553 8141 tablet_replica.cc:442] T e6913f76f6c64a438981736c7074d81a P 1d4dabac174c4e2bbc75a10c2fa03a4c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 04:07:08.384847 8176 tablet_replica.cc:439] T e6913f76f6c64a438981736c7074d81a P 5e50b5e8a58546bdb6509bba0687350b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5e50b5e8a58546bdb6509bba0687350b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d4dabac174c4e2bbc75a10c2fa03a4c" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 46371 } } peers { permanent_uuid: "f000d641e9094650bbd16cf6b7a35069" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 44589 } } peers { permanent_uuid: "5e50b5e8a58546bdb6509bba0687350b" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 36469 } } }
I20260810 04:07:08.384932 8176 tablet_replica.cc:442] T e6913f76f6c64a438981736c7074d81a P 5e50b5e8a58546bdb6509bba0687350b: This TxnStatusTablet replica's current role is: LEADER
I20260810 04:07:08.385251 8155 tablet_replica.cc:439] T e6913f76f6c64a438981736c7074d81a P f000d641e9094650bbd16cf6b7a35069: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5e50b5e8a58546bdb6509bba0687350b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d4dabac174c4e2bbc75a10c2fa03a4c" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 46371 } } peers { permanent_uuid: "f000d641e9094650bbd16cf6b7a35069" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 44589 } } peers { permanent_uuid: "5e50b5e8a58546bdb6509bba0687350b" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 36469 } } }
I20260810 04:07:08.385332 8155 tablet_replica.cc:442] T e6913f76f6c64a438981736c7074d81a P f000d641e9094650bbd16cf6b7a35069: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 04:07:08.390419 8129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:08.387946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.2:53867 (local address 127.31.32.3:36469)
0810 04:07:08.388019 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0810 04:07:08.388021 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:07:08.388148 (+ 127us) server_negotiation.cc:407] Connection header received
0810 04:07:08.388189 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:07:08.388190 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:07:08.388199 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:07:08.388279 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 04:07:08.388384 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:08.388806 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:08.389828 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:08.390020 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:08.390068 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:08.390240 (+ 172us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 04:07:08.390262 (+ 22us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 04:07:08.390294 (+ 32us) server_negotiation.cc:1035] Waiting for connection context
0810 04:07:08.390367 (+ 73us) server_negotiation.cc:299] Negotiation successful
0810 04:07:08.390380 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 04:07:08.390482 8186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:08.387866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.3:36469 (local address 127.31.32.2:53867)
0810 04:07:08.388032 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0810 04:07:08.388043 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 04:07:08.388106 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:07:08.388284 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:07:08.388284 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:07:08.388286 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 04:07:08.388327 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:08.388329 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:08.388874 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:08.388875 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:08.389750 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:08.389752 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:08.390088 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:08.390089 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:08.390147 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:08.390161 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 04:07:08.390314 (+ 153us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 04:07:08.390316 (+ 2us) client_negotiation.cc:769] Sending connection context
0810 04:07:08.390365 (+ 49us) client_negotiation.cc:240] Negotiation successful
0810 04:07:08.390405 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":102,"threads_started":1}
I20260810 04:07:08.392428 8051 raft_consensus.cc:1261] T 2f70d875744c4a8ba3f0ccfdff391772 P 5e50b5e8a58546bdb6509bba0687350b [term 1 FOLLOWER]: Refusing update from remote peer f000d641e9094650bbd16cf6b7a35069: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:07:08.392458 7741 raft_consensus.cc:1261] T 2f70d875744c4a8ba3f0ccfdff391772 P 1d4dabac174c4e2bbc75a10c2fa03a4c [term 1 FOLLOWER]: Refusing update from remote peer f000d641e9094650bbd16cf6b7a35069: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:07:08.392688 8144 consensus_queue.cc:1035] T 2f70d875744c4a8ba3f0ccfdff391772 P f000d641e9094650bbd16cf6b7a35069 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d4dabac174c4e2bbc75a10c2fa03a4c" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 46371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:07:08.392804 8155 consensus_queue.cc:1035] T 2f70d875744c4a8ba3f0ccfdff391772 P f000d641e9094650bbd16cf6b7a35069 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e50b5e8a58546bdb6509bba0687350b" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 36469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:07:08.401253 8191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:08.398298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.2:44589 (local address 127.31.32.3:44769)
0810 04:07:08.398525 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0810 04:07:08.398533 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0810 04:07:08.398581 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:07:08.398688 (+ 107us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:07:08.398689 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:07:08.398690 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 04:07:08.398730 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:08.398731 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:08.399584 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:08.399585 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:08.400459 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:08.400462 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:08.400894 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:08.400895 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:08.400948 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:08.400960 (+ 12us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 04:07:08.401125 (+ 165us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 04:07:08.401126 (+ 1us) client_negotiation.cc:769] Sending connection context
0810 04:07:08.401160 (+ 34us) client_negotiation.cc:240] Negotiation successful
0810 04:07:08.401188 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":97,"threads_started":1}
I20260810 04:07:08.401247 8132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:08.398444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.3:44769 (local address 127.31.32.2:44589)
0810 04:07:08.398486 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0810 04:07:08.398487 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 04:07:08.398587 (+ 100us) server_negotiation.cc:407] Connection header received
0810 04:07:08.398614 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:07:08.398615 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:07:08.398626 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:07:08.398689 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 04:07:08.398765 (+ 76us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:08.399510 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:08.400527 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:08.400821 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:08.400889 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:08.401031 (+ 142us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 04:07:08.401064 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 04:07:08.401115 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0810 04:07:08.401162 (+ 47us) server_negotiation.cc:299] Negotiation successful
0810 04:07:08.401181 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260810 04:07:08.409751 7568 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44942:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 04:07:08.409861 7568 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44942:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 04:07:08.410486 7568 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2c147e41e15f4984b013b451708c65b3: Sending DeleteTablet for 3 replicas of tablet 2f70d875744c4a8ba3f0ccfdff391772
I20260810 04:07:08.410725 8028 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f70d875744c4a8ba3f0ccfdff391772 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:07:08 UTC) from {username='oryx', principal='oryx/127.31.32.62@KRBTEST.COM'} at 127.0.0.1:36476
I20260810 04:07:08.410748 7875 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f70d875744c4a8ba3f0ccfdff391772 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:07:08 UTC) from {username='oryx', principal='oryx/127.31.32.62@KRBTEST.COM'} at 127.0.0.1:54646
I20260810 04:07:08.410827 8160 tablet_replica.cc:330] T 2f70d875744c4a8ba3f0ccfdff391772 P 5e50b5e8a58546bdb6509bba0687350b: stopping tablet replica
I20260810 04:07:08.410843 8161 tablet_replica.cc:330] T 2f70d875744c4a8ba3f0ccfdff391772 P f000d641e9094650bbd16cf6b7a35069: stopping tablet replica
I20260810 04:07:08.410880 8160 raft_consensus.cc:2229] T 2f70d875744c4a8ba3f0ccfdff391772 P 5e50b5e8a58546bdb6509bba0687350b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 04:07:08.410892 8161 raft_consensus.cc:2229] T 2f70d875744c4a8ba3f0ccfdff391772 P f000d641e9094650bbd16cf6b7a35069 [term 1 LEADER]: Raft consensus shutting down.
I20260810 04:07:08.410912 8160 raft_consensus.cc:2258] T 2f70d875744c4a8ba3f0ccfdff391772 P 5e50b5e8a58546bdb6509bba0687350b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:07:08.410952 8161 raft_consensus.cc:2258] T 2f70d875744c4a8ba3f0ccfdff391772 P f000d641e9094650bbd16cf6b7a35069 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:07:08.411206 8160 ts_tablet_manager.cc:1905] T 2f70d875744c4a8ba3f0ccfdff391772 P 5e50b5e8a58546bdb6509bba0687350b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:07:08.411300 8161 ts_tablet_manager.cc:1905] T 2f70d875744c4a8ba3f0ccfdff391772 P f000d641e9094650bbd16cf6b7a35069: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:07:08.412721 7720 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f70d875744c4a8ba3f0ccfdff391772 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:07:08 UTC) from {username='oryx', principal='oryx/127.31.32.62@KRBTEST.COM'} at 127.0.0.1:35138
I20260810 04:07:08.412853 8159 tablet_replica.cc:330] T 2f70d875744c4a8ba3f0ccfdff391772 P 1d4dabac174c4e2bbc75a10c2fa03a4c: stopping tablet replica
I20260810 04:07:08.412922 8159 raft_consensus.cc:2229] T 2f70d875744c4a8ba3f0ccfdff391772 P 1d4dabac174c4e2bbc75a10c2fa03a4c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 04:07:08.412943 8161 ts_tablet_manager.cc:1918] T 2f70d875744c4a8ba3f0ccfdff391772 P f000d641e9094650bbd16cf6b7a35069: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 04:07:08.412978 8159 raft_consensus.cc:2258] T 2f70d875744c4a8ba3f0ccfdff391772 P 1d4dabac174c4e2bbc75a10c2fa03a4c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:07:08.412981 8160 ts_tablet_manager.cc:1918] T 2f70d875744c4a8ba3f0ccfdff391772 P 5e50b5e8a58546bdb6509bba0687350b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 04:07:08.413017 8161 log.cc:1196] T 2f70d875744c4a8ba3f0ccfdff391772 P f000d641e9094650bbd16cf6b7a35069: Deleting WAL directory at /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786334794676589-31872-0/minicluster-data/ts-1/wal/wals/2f70d875744c4a8ba3f0ccfdff391772
I20260810 04:07:08.413033 8160 log.cc:1196] T 2f70d875744c4a8ba3f0ccfdff391772 P 5e50b5e8a58546bdb6509bba0687350b: Deleting WAL directory at /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786334794676589-31872-0/minicluster-data/ts-2/wal/wals/2f70d875744c4a8ba3f0ccfdff391772
I20260810 04:07:08.413275 8161 ts_tablet_manager.cc:1939] T 2f70d875744c4a8ba3f0ccfdff391772 P f000d641e9094650bbd16cf6b7a35069: Deleting consensus metadata
I20260810 04:07:08.413285 8160 ts_tablet_manager.cc:1939] T 2f70d875744c4a8ba3f0ccfdff391772 P 5e50b5e8a58546bdb6509bba0687350b: Deleting consensus metadata
I20260810 04:07:08.413674 7554 catalog_manager.cc:4841] TS 5e50b5e8a58546bdb6509bba0687350b (127.31.32.3:36469): tablet 2f70d875744c4a8ba3f0ccfdff391772 (table test-table [id=90be5f4aa8f44ae7bda5f0464461f01e]) successfully deleted
I20260810 04:07:08.413740 7553 catalog_manager.cc:4841] TS f000d641e9094650bbd16cf6b7a35069 (127.31.32.2:44589): tablet 2f70d875744c4a8ba3f0ccfdff391772 (table test-table [id=90be5f4aa8f44ae7bda5f0464461f01e]) successfully deleted
I20260810 04:07:08.413969 8159 ts_tablet_manager.cc:1905] T 2f70d875744c4a8ba3f0ccfdff391772 P 1d4dabac174c4e2bbc75a10c2fa03a4c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:07:08.415512 8159 ts_tablet_manager.cc:1918] T 2f70d875744c4a8ba3f0ccfdff391772 P 1d4dabac174c4e2bbc75a10c2fa03a4c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 04:07:08.415573 8159 log.cc:1196] T 2f70d875744c4a8ba3f0ccfdff391772 P 1d4dabac174c4e2bbc75a10c2fa03a4c: Deleting WAL directory at /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786334794676589-31872-0/minicluster-data/ts-0/wal/wals/2f70d875744c4a8ba3f0ccfdff391772
I20260810 04:07:08.415779 8159 ts_tablet_manager.cc:1939] T 2f70d875744c4a8ba3f0ccfdff391772 P 1d4dabac174c4e2bbc75a10c2fa03a4c: Deleting consensus metadata
I20260810 04:07:08.416234 7553 catalog_manager.cc:4841] TS 1d4dabac174c4e2bbc75a10c2fa03a4c (127.31.32.1:46371): tablet 2f70d875744c4a8ba3f0ccfdff391772 (table test-table [id=90be5f4aa8f44ae7bda5f0464461f01e]) successfully deleted
Aug 10 04:07:08 dist-test-slave-8w9v krb5kdc[7520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334828, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 10 04:07:08 dist-test-slave-8w9v krb5kdc[7520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.31.32.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 04:07:08 dist-test-slave-8w9v krb5kdc[7520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.31.32.62@KRBTEST.COM, Server not found in Kerberos database
W20260810 04:07:08.440258 7621 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:07:08.433950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44966 (local address 127.31.32.62:39225)
0810 04:07:08.434046 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0810 04:07:08.434049 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:07:08.434208 (+ 159us) server_negotiation.cc:407] Connection header received
0810 04:07:08.434287 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:07:08.434287 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:07:08.434299 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:07:08.434382 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:07:08.435120 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:08.435631 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:08.436787 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:08.436976 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:08.437076 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:08.440190 (+ 3114us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44966: BlockingRecv error: recv got EOF from 127.0.0.1:44966 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Aug 10 04:07:08 dist-test-slave-8w9v krb5kdc[7520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334828, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.31.32.62@KRBTEST.COM
I20260810 04:07:08.448799 7621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:08.441302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44974 (local address 127.31.32.62:39225)
0810 04:07:08.441343 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0810 04:07:08.441345 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:07:08.441421 (+ 76us) server_negotiation.cc:407] Connection header received
0810 04:07:08.441614 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:07:08.441615 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:07:08.441642 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:07:08.441700 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:07:08.442218 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:08.442806 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:08.443547 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:08.443737 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:08.443844 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:08.446144 (+ 2300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:07:08.446146 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:07:08.446147 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:07:08.446154 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:07:08.447555 (+ 1401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:07:08.447948 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:07:08.447950 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:07:08.447950 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:07:08.448015 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:07:08.448261 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:07:08.448262 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:07:08.448262 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:07:08.448379 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:07:08.448504 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0810 04:07:08.448616 (+ 112us) server_negotiation.cc:299] Negotiation successful
0810 04:07:08.448694 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260810 04:07:08.450469 7568 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44974:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 04:07:08.450757 7568 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 04:07:08.452270 7720 tablet_service.cc:1467] Processing CreateTablet for tablet a1647704752e411db3f7d9f867433e8d (DEFAULT_TABLE table=test-table [id=27e62ea481c444349a81b6961bc5b436]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:07:08.452323 7875 tablet_service.cc:1467] Processing CreateTablet for tablet a1647704752e411db3f7d9f867433e8d (DEFAULT_TABLE table=test-table [id=27e62ea481c444349a81b6961bc5b436]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:07:08.452414 7720 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1647704752e411db3f7d9f867433e8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:07:08.452495 7875 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1647704752e411db3f7d9f867433e8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:07:08.453583 8135 tablet_bootstrap.cc:492] T a1647704752e411db3f7d9f867433e8d P f000d641e9094650bbd16cf6b7a35069: Bootstrap starting.
I20260810 04:07:08.453924 8134 tablet_bootstrap.cc:492] T a1647704752e411db3f7d9f867433e8d P 1d4dabac174c4e2bbc75a10c2fa03a4c: Bootstrap starting.
I20260810 04:07:08.453954 8135 tablet_bootstrap.cc:654] T a1647704752e411db3f7d9f867433e8d P f000d641e9094650bbd16cf6b7a35069: Neither blocks nor log segments found. Creating new log.
I20260810 04:07:08.453948 8028 tablet_service.cc:1467] Processing CreateTablet for tablet a1647704752e411db3f7d9f867433e8d (DEFAULT_TABLE table=test-table [id=27e62ea481c444349a81b6961bc5b436]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:07:08.454074 8028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1647704752e411db3f7d9f867433e8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:07:08.454396 8134 tablet_bootstrap.cc:654] T a1647704752e411db3f7d9f867433e8d P 1d4dabac174c4e2bbc75a10c2fa03a4c: Neither blocks nor log segments found. Creating new log.
I20260810 04:07:08.455121 8134 tablet_bootstrap.cc:492] T a1647704752e411db3f7d9f867433e8d P 1d4dabac174c4e2bbc75a10c2fa03a4c: No bootstrap required, opened a new log
I20260810 04:07:08.455197 8134 ts_tablet_manager.cc:1397] T a1647704752e411db3f7d9f867433e8d P 1d4dabac174c4e2bbc75a10c2fa03a4c: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260810 04:07:08.455351 8133 tablet_bootstrap.cc:492] T a1647704752e411db3f7d9f867433e8d P 5e50b5e8a58546bdb6509bba0687350b: Bootstrap starting.
I20260810 04:07:08.455384 8134 raft_consensus.cc:348] T a1647704752e411db3f7d9f867433e8d P 1d4dabac174c4e2bbc75a10c2fa03a4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d4dabac174c4e2bbc75a10c2fa03a4c" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 46371 } } peers { permanent_uuid: "f000d641e9094650bbd16cf6b7a35069" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 44589 } } peers { permanent_uuid: "5e50b5e8a58546bdb6509bba0687350b" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 36469 } }
I20260810 04:07:08.455494 8134 raft_consensus.cc:374] T a1647704752e411db3f7d9f867433e8d P 1d4dabac174c4e2bbc75a10c2fa03a4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:07:08.455526 8134 raft_consensus.cc:729] T a1647704752e411db3f7d9f867433e8d P 1d4dabac174c4e2bbc75a10c2fa03a4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d4dabac174c4e2bbc75a10c2fa03a4c, State: Initialized, Role: FOLLOWER
I20260810 04:07:08.455575 8134 consensus_queue.cc:260] T a1647704752e411db3f7d9f867433e8d P 1d4dabac174c4e2bbc75a10c2fa03a4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d4dabac174c4e2bbc75a10c2fa03a4c" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 46371 } } peers { permanent_uuid: "f000d641e9094650bbd16cf6b7a35069" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 44589 } } peers { permanent_uuid: "5e50b5e8a58546bdb6509bba0687350b" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 36469 } }
I20260810 04:07:08.455667 8133 tablet_bootstrap.cc:654] T a1647704752e411db3f7d9f867433e8d P 5e50b5e8a58546bdb6509bba0687350b: Neither blocks nor log segments found. Creating new log.
I20260810 04:07:08.455675 8134 ts_tablet_manager.cc:1428] T a1647704752e411db3f7d9f867433e8d P 1d4dabac174c4e2bbc75a10c2fa03a4c: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260810 04:07:08.456305 8133 tablet_bootstrap.cc:492] T a1647704752e411db3f7d9f867433e8d P 5e50b5e8a58546bdb6509bba0687350b: No bootstrap required, opened a new log
I20260810 04:07:08.456362 8133 ts_tablet_manager.cc:1397] T a1647704752e411db3f7d9f867433e8d P 5e50b5e8a58546bdb6509bba0687350b: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260810 04:07:08.456504 8135 tablet_bootstrap.cc:492] T a1647704752e411db3f7d9f867433e8d P f000d641e9094650bbd16cf6b7a35069: No bootstrap required, opened a new log
I20260810 04:07:08.456499 8133 raft_consensus.cc:348] T a1647704752e411db3f7d9f867433e8d P 5e50b5e8a58546bdb6509bba0687350b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d4dabac174c4e2bbc75a10c2fa03a4c" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 46371 } } peers { permanent_uuid: "f000d641e9094650bbd16cf6b7a35069" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 44589 } } peers { permanent_uuid: "5e50b5e8a58546bdb6509bba0687350b" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 36469 } }
I20260810 04:07:08.456552 8133 raft_consensus.cc:374] T a1647704752e411db3f7d9f867433e8d P 5e50b5e8a58546bdb6509bba0687350b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:07:08.456550 8135 ts_tablet_manager.cc:1397] T a1647704752e411db3f7d9f867433e8d P f000d641e9094650bbd16cf6b7a35069: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260810 04:07:08.456569 8133 raft_consensus.cc:729] T a1647704752e411db3f7d9f867433e8d P 5e50b5e8a58546bdb6509bba0687350b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e50b5e8a58546bdb6509bba0687350b, State: Initialized, Role: FOLLOWER
I20260810 04:07:08.456630 8133 consensus_queue.cc:260] T a1647704752e411db3f7d9f867433e8d P 5e50b5e8a58546bdb6509bba0687350b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d4dabac174c4e2bbc75a10c2fa03a4c" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 46371 } } peers { permanent_uuid: "f000d641e9094650bbd16cf6b7a35069" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 44589 } } peers { permanent_uuid: "5e50b5e8a58546bdb6509bba0687350b" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 36469 } }
I20260810 04:07:08.456664 8135 raft_consensus.cc:348] T a1647704752e411db3f7d9f867433e8d P f000d641e9094650bbd16cf6b7a35069 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d4dabac174c4e2bbc75a10c2fa03a4c" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 46371 } } peers { permanent_uuid: "f000d641e9094650bbd16cf6b7a35069" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 44589 } } peers { permanent_uuid: "5e50b5e8a58546bdb6509bba0687350b" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 36469 } }
I20260810 04:07:08.456733 8135 raft_consensus.cc:374] T a1647704752e411db3f7d9f867433e8d P f000d641e9094650bbd16cf6b7a35069 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:07:08.456758 8135 raft_consensus.cc:729] T a1647704752e411db3f7d9f867433e8d P f000d641e9094650bbd16cf6b7a35069 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f000d641e9094650bbd16cf6b7a35069, State: Initialized, Role: FOLLOWER
I20260810 04:07:08.456773 8133 ts_tablet_manager.cc:1428] T a1647704752e411db3f7d9f867433e8d P 5e50b5e8a58546bdb6509bba0687350b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260810 04:07:08.456805 8135 consensus_queue.cc:260] T a1647704752e411db3f7d9f867433e8d P f000d641e9094650bbd16cf6b7a35069 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d4dabac174c4e2bbc75a10c2fa03a4c" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 46371 } } peers { permanent_uuid: "f000d641e9094650bbd16cf6b7a35069" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 44589 } } peers { permanent_uuid: "5e50b5e8a58546bdb6509bba0687350b" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 36469 } }
I20260810 04:07:08.456912 8135 ts_tablet_manager.cc:1428] T a1647704752e411db3f7d9f867433e8d P f000d641e9094650bbd16cf6b7a35069: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260810 04:07:08.474997 8141 raft_consensus.cc:482] T a1647704752e411db3f7d9f867433e8d P 1d4dabac174c4e2bbc75a10c2fa03a4c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 04:07:08.475091 8141 raft_consensus.cc:504] T a1647704752e411db3f7d9f867433e8d P 1d4dabac174c4e2bbc75a10c2fa03a4c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d4dabac174c4e2bbc75a10c2fa03a4c" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 46371 } } peers { permanent_uuid: "f000d641e9094650bbd16cf6b7a35069" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 44589 } } peers { permanent_uuid: "5e50b5e8a58546bdb6509bba0687350b" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 36469 } }
I20260810 04:07:08.475528 8141 leader_election.cc:288] T a1647704752e411db3f7d9f867433e8d P 1d4dabac174c4e2bbc75a10c2fa03a4c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f000d641e9094650bbd16cf6b7a35069 (127.31.32.2:44589), 5e50b5e8a58546bdb6509bba0687350b (127.31.32.3:36469)
I20260810 04:07:08.479386 8132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:08.475891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.1:55123 (local address 127.31.32.2:44589)
0810 04:07:08.475935 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0810 04:07:08.475937 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:07:08.475984 (+ 47us) server_negotiation.cc:407] Connection header received
0810 04:07:08.476108 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:07:08.476109 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:07:08.476153 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:07:08.476184 (+ 31us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:07:08.476384 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:08.477183 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:08.478605 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:08.479017 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:08.479102 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:08.479114 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0810 04:07:08.479277 (+ 163us) server_negotiation.cc:299] Negotiation successful
0810 04:07:08.479322 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260810 04:07:08.479399 8212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:08.475696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.2:44589 (local address 127.31.32.1:55123)
0810 04:07:08.475926 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0810 04:07:08.475935 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 04:07:08.476029 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:07:08.476245 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:07:08.476245 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:07:08.476246 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:07:08.476302 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:08.476303 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:08.477309 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:08.477311 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:08.478509 (+ 1198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:08.478513 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:08.479103 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:08.479104 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:08.479183 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:08.479187 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 04:07:08.479267 (+ 80us) client_negotiation.cc:240] Negotiation successful
0810 04:07:08.479307 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":88,"threads_started":1}
I20260810 04:07:08.479662 8129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:08.475824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.1:44179 (local address 127.31.32.3:36469)
0810 04:07:08.475869 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0810 04:07:08.475872 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:07:08.475935 (+ 63us) server_negotiation.cc:407] Connection header received
0810 04:07:08.476067 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:07:08.476068 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:07:08.476079 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:07:08.476109 (+ 30us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:07:08.476436 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:08.477031 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:08.478713 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:08.479282 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:08.479357 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:08.479368 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0810 04:07:08.479569 (+ 201us) server_negotiation.cc:299] Negotiation successful
0810 04:07:08.479606 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260810 04:07:08.479683 8210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:08.475677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.3:36469 (local address 127.31.32.1:44179)
0810 04:07:08.475866 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0810 04:07:08.475879 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 04:07:08.475965 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:07:08.476203 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:07:08.476204 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:07:08.476205 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:07:08.476358 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:08.476360 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:08.477130 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:08.477132 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:08.478546 (+ 1414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:08.478550 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:08.479426 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:08.479427 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:08.479496 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:08.479500 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 04:07:08.479560 (+ 60us) client_negotiation.cc:240] Negotiation successful
0810 04:07:08.479600 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":76,"threads_started":1}
I20260810 04:07:08.480048 7899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1647704752e411db3f7d9f867433e8d" candidate_uuid: "1d4dabac174c4e2bbc75a10c2fa03a4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f000d641e9094650bbd16cf6b7a35069" is_pre_election: true
I20260810 04:07:08.480170 7899 raft_consensus.cc:2454] T a1647704752e411db3f7d9f867433e8d P f000d641e9094650bbd16cf6b7a35069 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d4dabac174c4e2bbc75a10c2fa03a4c in term 0.
I20260810 04:07:08.480113 8051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1647704752e411db3f7d9f867433e8d" candidate_uuid: "1d4dabac174c4e2bbc75a10c2fa03a4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e50b5e8a58546bdb6509bba0687350b" is_pre_election: true
I20260810 04:07:08.480221 8051 raft_consensus.cc:2454] T a1647704752e411db3f7d9f867433e8d P 5e50b5e8a58546bdb6509bba0687350b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d4dabac174c4e2bbc75a10c2fa03a4c in term 0.
I20260810 04:07:08.480356 7670 leader_election.cc:302] T a1647704752e411db3f7d9f867433e8d P 1d4dabac174c4e2bbc75a10c2fa03a4c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d4dabac174c4e2bbc75a10c2fa03a4c, f000d641e9094650bbd16cf6b7a35069; no voters:
I20260810 04:07:08.480513 8141 raft_consensus.cc:2781] T a1647704752e411db3f7d9f867433e8d P 1d4dabac174c4e2bbc75a10c2fa03a4c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 04:07:08.480551 8141 raft_consensus.cc:482] T a1647704752e411db3f7d9f867433e8d P 1d4dabac174c4e2bbc75a10c2fa03a4c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 04:07:08.480587 8141 raft_consensus.cc:3037] T a1647704752e411db3f7d9f867433e8d P 1d4dabac174c4e2bbc75a10c2fa03a4c [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:07:08.481297 8141 raft_consensus.cc:504] T a1647704752e411db3f7d9f867433e8d P 1d4dabac174c4e2bbc75a10c2fa03a4c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d4dabac174c4e2bbc75a10c2fa03a4c" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 46371 } } peers { permanent_uuid: "f000d641e9094650bbd16cf6b7a35069" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 44589 } } peers { permanent_uuid: "5e50b5e8a58546bdb6509bba0687350b" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 36469 } }
I20260810 04:07:08.481447 8141 leader_election.cc:288] T a1647704752e411db3f7d9f867433e8d P 1d4dabac174c4e2bbc75a10c2fa03a4c [CANDIDATE]: Term 1 election: Requested vote from peers f000d641e9094650bbd16cf6b7a35069 (127.31.32.2:44589), 5e50b5e8a58546bdb6509bba0687350b (127.31.32.3:36469)
I20260810 04:07:08.481575 7899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1647704752e411db3f7d9f867433e8d" candidate_uuid: "1d4dabac174c4e2bbc75a10c2fa03a4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f000d641e9094650bbd16cf6b7a35069"
I20260810 04:07:08.481658 7899 raft_consensus.cc:3037] T a1647704752e411db3f7d9f867433e8d P f000d641e9094650bbd16cf6b7a35069 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:07:08.481635 8051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1647704752e411db3f7d9f867433e8d" candidate_uuid: "1d4dabac174c4e2bbc75a10c2fa03a4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e50b5e8a58546bdb6509bba0687350b"
I20260810 04:07:08.481724 8051 raft_consensus.cc:3037] T a1647704752e411db3f7d9f867433e8d P 5e50b5e8a58546bdb6509bba0687350b [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:07:08.482317 8051 raft_consensus.cc:2454] T a1647704752e411db3f7d9f867433e8d P 5e50b5e8a58546bdb6509bba0687350b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d4dabac174c4e2bbc75a10c2fa03a4c in term 1.
I20260810 04:07:08.482491 7671 leader_election.cc:302] T a1647704752e411db3f7d9f867433e8d P 1d4dabac174c4e2bbc75a10c2fa03a4c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d4dabac174c4e2bbc75a10c2fa03a4c, 5e50b5e8a58546bdb6509bba0687350b; no voters:
I20260810 04:07:08.482533 7899 raft_consensus.cc:2454] T a1647704752e411db3f7d9f867433e8d P f000d641e9094650bbd16cf6b7a35069 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d4dabac174c4e2bbc75a10c2fa03a4c in term 1.
I20260810 04:07:08.482604 8141 raft_consensus.cc:2781] T a1647704752e411db3f7d9f867433e8d P 1d4dabac174c4e2bbc75a10c2fa03a4c [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:07:08.482764 8141 raft_consensus.cc:686] T a1647704752e411db3f7d9f867433e8d P 1d4dabac174c4e2bbc75a10c2fa03a4c [term 1 LEADER]: Becoming Leader. State: Replica: 1d4dabac174c4e2bbc75a10c2fa03a4c, State: Running, Role: LEADER
I20260810 04:07:08.482842 8141 consensus_queue.cc:237] T a1647704752e411db3f7d9f867433e8d P 1d4dabac174c4e2bbc75a10c2fa03a4c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d4dabac174c4e2bbc75a10c2fa03a4c" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 46371 } } peers { permanent_uuid: "f000d641e9094650bbd16cf6b7a35069" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 44589 } } peers { permanent_uuid: "5e50b5e8a58546bdb6509bba0687350b" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 36469 } }
I20260810 04:07:08.483580 7566 catalog_manager.cc:5495] T a1647704752e411db3f7d9f867433e8d P 1d4dabac174c4e2bbc75a10c2fa03a4c reported cstate change: term changed from 0 to 1, leader changed from <none> to 1d4dabac174c4e2bbc75a10c2fa03a4c (127.31.32.1). New cstate: current_term: 1 leader_uuid: "1d4dabac174c4e2bbc75a10c2fa03a4c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d4dabac174c4e2bbc75a10c2fa03a4c" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 46371 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f000d641e9094650bbd16cf6b7a35069" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 44589 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e50b5e8a58546bdb6509bba0687350b" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 36469 } health_report { overall_health: UNKNOWN } } }
I20260810 04:07:08.494436 8131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:08.492053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35140 (local address 127.31.32.1:46371)
0810 04:07:08.492101 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0810 04:07:08.492103 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:07:08.492114 (+ 11us) server_negotiation.cc:407] Connection header received
0810 04:07:08.492168 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:07:08.492169 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:07:08.492188 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:07:08.492274 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 04:07:08.492423 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:08.492872 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:08.493646 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:08.493841 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:08.493922 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:08.494113 (+ 191us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 04:07:08.494213 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 04:07:08.494286 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0810 04:07:08.494342 (+ 56us) server_negotiation.cc:299] Negotiation successful
0810 04:07:08.494360 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260810 04:07:08.495405 8051 raft_consensus.cc:1261] T a1647704752e411db3f7d9f867433e8d P 5e50b5e8a58546bdb6509bba0687350b [term 1 FOLLOWER]: Refusing update from remote peer 1d4dabac174c4e2bbc75a10c2fa03a4c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:07:08.495518 7899 raft_consensus.cc:1261] T a1647704752e411db3f7d9f867433e8d P f000d641e9094650bbd16cf6b7a35069 [term 1 FOLLOWER]: Refusing update from remote peer 1d4dabac174c4e2bbc75a10c2fa03a4c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:07:08.495692 8213 consensus_queue.cc:1035] T a1647704752e411db3f7d9f867433e8d P 1d4dabac174c4e2bbc75a10c2fa03a4c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e50b5e8a58546bdb6509bba0687350b" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 36469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:07:08.495798 8141 consensus_queue.cc:1035] T a1647704752e411db3f7d9f867433e8d P 1d4dabac174c4e2bbc75a10c2fa03a4c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f000d641e9094650bbd16cf6b7a35069" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 44589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:07:08.499447 7568 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44974:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 04:07:08.499538 7568 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44974:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 04:07:08.500141 7568 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2c147e41e15f4984b013b451708c65b3: Sending DeleteTablet for 3 replicas of tablet a1647704752e411db3f7d9f867433e8d
I20260810 04:07:08.500408 7875 tablet_service.cc:1514] Processing DeleteTablet for tablet a1647704752e411db3f7d9f867433e8d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:07:08 UTC) from {username='oryx', principal='oryx/127.31.32.62@KRBTEST.COM'} at 127.0.0.1:54646
I20260810 04:07:08.500504 8161 tablet_replica.cc:330] T a1647704752e411db3f7d9f867433e8d P f000d641e9094650bbd16cf6b7a35069: stopping tablet replica
I20260810 04:07:08.500556 8161 raft_consensus.cc:2229] T a1647704752e411db3f7d9f867433e8d P f000d641e9094650bbd16cf6b7a35069 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 04:07:08.500542 8028 tablet_service.cc:1514] Processing DeleteTablet for tablet a1647704752e411db3f7d9f867433e8d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:07:08 UTC) from {username='oryx', principal='oryx/127.31.32.62@KRBTEST.COM'} at 127.0.0.1:36476
I20260810 04:07:08.500597 8161 raft_consensus.cc:2258] T a1647704752e411db3f7d9f867433e8d P f000d641e9094650bbd16cf6b7a35069 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:07:08.500628 8160 tablet_replica.cc:330] T a1647704752e411db3f7d9f867433e8d P 5e50b5e8a58546bdb6509bba0687350b: stopping tablet replica
I20260810 04:07:08.500669 8160 raft_consensus.cc:2229] T a1647704752e411db3f7d9f867433e8d P 5e50b5e8a58546bdb6509bba0687350b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 04:07:08.500710 8160 raft_consensus.cc:2258] T a1647704752e411db3f7d9f867433e8d P 5e50b5e8a58546bdb6509bba0687350b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:07:08.500783 7720 tablet_service.cc:1514] Processing DeleteTablet for tablet a1647704752e411db3f7d9f867433e8d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:07:08 UTC) from {username='oryx', principal='oryx/127.31.32.62@KRBTEST.COM'} at 127.0.0.1:35138
I20260810 04:07:08.500867 8159 tablet_replica.cc:330] T a1647704752e411db3f7d9f867433e8d P 1d4dabac174c4e2bbc75a10c2fa03a4c: stopping tablet replica
I20260810 04:07:08.500910 8159 raft_consensus.cc:2229] T a1647704752e411db3f7d9f867433e8d P 1d4dabac174c4e2bbc75a10c2fa03a4c [term 1 LEADER]: Raft consensus shutting down.
I20260810 04:07:08.500982 8159 raft_consensus.cc:2258] T a1647704752e411db3f7d9f867433e8d P 1d4dabac174c4e2bbc75a10c2fa03a4c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:07:08.501286 8159 ts_tablet_manager.cc:1905] T a1647704752e411db3f7d9f867433e8d P 1d4dabac174c4e2bbc75a10c2fa03a4c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:07:08.501472 7568 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44974:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 04:07:08.501611 7568 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 04:07:08.502331 8160 ts_tablet_manager.cc:1905] T a1647704752e411db3f7d9f867433e8d P 5e50b5e8a58546bdb6509bba0687350b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:07:08.503314 8161 ts_tablet_manager.cc:1905] T a1647704752e411db3f7d9f867433e8d P f000d641e9094650bbd16cf6b7a35069: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:07:08.503953 8160 ts_tablet_manager.cc:1918] T a1647704752e411db3f7d9f867433e8d P 5e50b5e8a58546bdb6509bba0687350b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 04:07:08.504010 8160 log.cc:1196] T a1647704752e411db3f7d9f867433e8d P 5e50b5e8a58546bdb6509bba0687350b: Deleting WAL directory at /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786334794676589-31872-0/minicluster-data/ts-2/wal/wals/a1647704752e411db3f7d9f867433e8d
I20260810 04:07:08.504249 8160 ts_tablet_manager.cc:1939] T a1647704752e411db3f7d9f867433e8d P 5e50b5e8a58546bdb6509bba0687350b: Deleting consensus metadata
I20260810 04:07:08.504690 7554 catalog_manager.cc:4841] TS 5e50b5e8a58546bdb6509bba0687350b (127.31.32.3:36469): tablet a1647704752e411db3f7d9f867433e8d (table test-table [id=27e62ea481c444349a81b6961bc5b436]) successfully deleted
I20260810 04:07:08.504820 8161 ts_tablet_manager.cc:1918] T a1647704752e411db3f7d9f867433e8d P f000d641e9094650bbd16cf6b7a35069: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 04:07:08.504889 8161 log.cc:1196] T a1647704752e411db3f7d9f867433e8d P f000d641e9094650bbd16cf6b7a35069: Deleting WAL directory at /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786334794676589-31872-0/minicluster-data/ts-1/wal/wals/a1647704752e411db3f7d9f867433e8d
I20260810 04:07:08.505108 8161 ts_tablet_manager.cc:1939] T a1647704752e411db3f7d9f867433e8d P f000d641e9094650bbd16cf6b7a35069: Deleting consensus metadata
I20260810 04:07:08.505438 7553 catalog_manager.cc:4841] TS f000d641e9094650bbd16cf6b7a35069 (127.31.32.2:44589): tablet a1647704752e411db3f7d9f867433e8d (table test-table [id=27e62ea481c444349a81b6961bc5b436]) successfully deleted
I20260810 04:07:08.506153 8028 tablet_service.cc:1467] Processing CreateTablet for tablet 354755d4f25a4771bdc3a321c4defef8 (DEFAULT_TABLE table=test-table [id=02bfb23f6c6a470abbc66d00e7c57963]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:07:08.506171 7720 tablet_service.cc:1467] Processing CreateTablet for tablet 354755d4f25a4771bdc3a321c4defef8 (DEFAULT_TABLE table=test-table [id=02bfb23f6c6a470abbc66d00e7c57963]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:07:08.506269 8028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 354755d4f25a4771bdc3a321c4defef8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:07:08.507536 8133 tablet_bootstrap.cc:492] T 354755d4f25a4771bdc3a321c4defef8 P 5e50b5e8a58546bdb6509bba0687350b: Bootstrap starting.
I20260810 04:07:08.507570 7875 tablet_service.cc:1467] Processing CreateTablet for tablet 354755d4f25a4771bdc3a321c4defef8 (DEFAULT_TABLE table=test-table [id=02bfb23f6c6a470abbc66d00e7c57963]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:07:08.507668 7875 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 354755d4f25a4771bdc3a321c4defef8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:07:08.507961 8133 tablet_bootstrap.cc:654] T 354755d4f25a4771bdc3a321c4defef8 P 5e50b5e8a58546bdb6509bba0687350b: Neither blocks nor log segments found. Creating new log.
I20260810 04:07:08.508970 8135 tablet_bootstrap.cc:492] T 354755d4f25a4771bdc3a321c4defef8 P f000d641e9094650bbd16cf6b7a35069: Bootstrap starting.
I20260810 04:07:08.509272 8135 tablet_bootstrap.cc:654] T 354755d4f25a4771bdc3a321c4defef8 P f000d641e9094650bbd16cf6b7a35069: Neither blocks nor log segments found. Creating new log.
I20260810 04:07:08.509322 8133 tablet_bootstrap.cc:492] T 354755d4f25a4771bdc3a321c4defef8 P 5e50b5e8a58546bdb6509bba0687350b: No bootstrap required, opened a new log
I20260810 04:07:08.509367 8133 ts_tablet_manager.cc:1397] T 354755d4f25a4771bdc3a321c4defef8 P 5e50b5e8a58546bdb6509bba0687350b: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260810 04:07:08.509514 8133 raft_consensus.cc:348] T 354755d4f25a4771bdc3a321c4defef8 P 5e50b5e8a58546bdb6509bba0687350b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d4dabac174c4e2bbc75a10c2fa03a4c" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 46371 } } peers { permanent_uuid: "f000d641e9094650bbd16cf6b7a35069" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 44589 } } peers { permanent_uuid: "5e50b5e8a58546bdb6509bba0687350b" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 36469 } }
I20260810 04:07:08.509580 8133 raft_consensus.cc:374] T 354755d4f25a4771bdc3a321c4defef8 P 5e50b5e8a58546bdb6509bba0687350b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:07:08.509599 8133 raft_consensus.cc:729] T 354755d4f25a4771bdc3a321c4defef8 P 5e50b5e8a58546bdb6509bba0687350b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e50b5e8a58546bdb6509bba0687350b, State: Initialized, Role: FOLLOWER
I20260810 04:07:08.509642 8133 consensus_queue.cc:260] T 354755d4f25a4771bdc3a321c4defef8 P 5e50b5e8a58546bdb6509bba0687350b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d4dabac174c4e2bbc75a10c2fa03a4c" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 46371 } } peers { permanent_uuid: "f000d641e9094650bbd16cf6b7a35069" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 44589 } } peers { permanent_uuid: "5e50b5e8a58546bdb6509bba0687350b" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 36469 } }
I20260810 04:07:08.509725 8133 ts_tablet_manager.cc:1428] T 354755d4f25a4771bdc3a321c4defef8 P 5e50b5e8a58546bdb6509bba0687350b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260810 04:07:08.509943 8159 ts_tablet_manager.cc:1918] T a1647704752e411db3f7d9f867433e8d P 1d4dabac174c4e2bbc75a10c2fa03a4c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 04:07:08.510003 8159 log.cc:1196] T a1647704752e411db3f7d9f867433e8d P 1d4dabac174c4e2bbc75a10c2fa03a4c: Deleting WAL directory at /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786334794676589-31872-0/minicluster-data/ts-0/wal/wals/a1647704752e411db3f7d9f867433e8d
I20260810 04:07:08.510071 7720 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 354755d4f25a4771bdc3a321c4defef8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:07:08.510223 8159 ts_tablet_manager.cc:1939] T a1647704752e411db3f7d9f867433e8d P 1d4dabac174c4e2bbc75a10c2fa03a4c: Deleting consensus metadata
I20260810 04:07:08.510771 8135 tablet_bootstrap.cc:492] T 354755d4f25a4771bdc3a321c4defef8 P f000d641e9094650bbd16cf6b7a35069: No bootstrap required, opened a new log
I20260810 04:07:08.510813 8135 ts_tablet_manager.cc:1397] T 354755d4f25a4771bdc3a321c4defef8 P f000d641e9094650bbd16cf6b7a35069: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 04:07:08.510914 8135 raft_consensus.cc:348] T 354755d4f25a4771bdc3a321c4defef8 P f000d641e9094650bbd16cf6b7a35069 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d4dabac174c4e2bbc75a10c2fa03a4c" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 46371 } } peers { permanent_uuid: "f000d641e9094650bbd16cf6b7a35069" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 44589 } } peers { permanent_uuid: "5e50b5e8a58546bdb6509bba0687350b" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 36469 } }
I20260810 04:07:08.510975 8135 raft_consensus.cc:374] T 354755d4f25a4771bdc3a321c4defef8 P f000d641e9094650bbd16cf6b7a35069 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:07:08.510994 8135 raft_consensus.cc:729] T 354755d4f25a4771bdc3a321c4defef8 P f000d641e9094650bbd16cf6b7a35069 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f000d641e9094650bbd16cf6b7a35069, State: Initialized, Role: FOLLOWER
I20260810 04:07:08.511031 8135 consensus_queue.cc:260] T 354755d4f25a4771bdc3a321c4defef8 P f000d641e9094650bbd16cf6b7a35069 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d4dabac174c4e2bbc75a10c2fa03a4c" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 46371 } } peers { permanent_uuid: "f000d641e9094650bbd16cf6b7a35069" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 44589 } } peers { permanent_uuid: "5e50b5e8a58546bdb6509bba0687350b" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 36469 } }
I20260810 04:07:08.511127 8135 ts_tablet_manager.cc:1428] T 354755d4f25a4771bdc3a321c4defef8 P f000d641e9094650bbd16cf6b7a35069: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260810 04:07:08.511493 8134 tablet_bootstrap.cc:492] T 354755d4f25a4771bdc3a321c4defef8 P 1d4dabac174c4e2bbc75a10c2fa03a4c: Bootstrap starting.
I20260810 04:07:08.511890 8134 tablet_bootstrap.cc:654] T 354755d4f25a4771bdc3a321c4defef8 P 1d4dabac174c4e2bbc75a10c2fa03a4c: Neither blocks nor log segments found. Creating new log.
I20260810 04:07:08.512193 7553 catalog_manager.cc:4841] TS 1d4dabac174c4e2bbc75a10c2fa03a4c (127.31.32.1:46371): tablet a1647704752e411db3f7d9f867433e8d (table test-table [id=27e62ea481c444349a81b6961bc5b436]) successfully deleted
I20260810 04:07:08.512560 8134 tablet_bootstrap.cc:492] T 354755d4f25a4771bdc3a321c4defef8 P 1d4dabac174c4e2bbc75a10c2fa03a4c: No bootstrap required, opened a new log
I20260810 04:07:08.512615 8134 ts_tablet_manager.cc:1397] T 354755d4f25a4771bdc3a321c4defef8 P 1d4dabac174c4e2bbc75a10c2fa03a4c: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260810 04:07:08.512753 8134 raft_consensus.cc:348] T 354755d4f25a4771bdc3a321c4defef8 P 1d4dabac174c4e2bbc75a10c2fa03a4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d4dabac174c4e2bbc75a10c2fa03a4c" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 46371 } } peers { permanent_uuid: "f000d641e9094650bbd16cf6b7a35069" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 44589 } } peers { permanent_uuid: "5e50b5e8a58546bdb6509bba0687350b" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 36469 } }
I20260810 04:07:08.512837 8134 raft_consensus.cc:374] T 354755d4f25a4771bdc3a321c4defef8 P 1d4dabac174c4e2bbc75a10c2fa03a4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:07:08.512859 8134 raft_consensus.cc:729] T 354755d4f25a4771bdc3a321c4defef8 P 1d4dabac174c4e2bbc75a10c2fa03a4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d4dabac174c4e2bbc75a10c2fa03a4c, State: Initialized, Role: FOLLOWER
I20260810 04:07:08.512905 8134 consensus_queue.cc:260] T 354755d4f25a4771bdc3a321c4defef8 P 1d4dabac174c4e2bbc75a10c2fa03a4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d4dabac174c4e2bbc75a10c2fa03a4c" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 46371 } } peers { permanent_uuid: "f000d641e9094650bbd16cf6b7a35069" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 44589 } } peers { permanent_uuid: "5e50b5e8a58546bdb6509bba0687350b" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 36469 } }
I20260810 04:07:08.512989 8134 ts_tablet_manager.cc:1428] T 354755d4f25a4771bdc3a321c4defef8 P 1d4dabac174c4e2bbc75a10c2fa03a4c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260810 04:07:08.694905 8213 raft_consensus.cc:482] T 354755d4f25a4771bdc3a321c4defef8 P 1d4dabac174c4e2bbc75a10c2fa03a4c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 04:07:08.695024 8213 raft_consensus.cc:504] T 354755d4f25a4771bdc3a321c4defef8 P 1d4dabac174c4e2bbc75a10c2fa03a4c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d4dabac174c4e2bbc75a10c2fa03a4c" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 46371 } } peers { permanent_uuid: "f000d641e9094650bbd16cf6b7a35069" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 44589 } } peers { permanent_uuid: "5e50b5e8a58546bdb6509bba0687350b" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 36469 } }
I20260810 04:07:08.695262 8213 leader_election.cc:288] T 354755d4f25a4771bdc3a321c4defef8 P 1d4dabac174c4e2bbc75a10c2fa03a4c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f000d641e9094650bbd16cf6b7a35069 (127.31.32.2:44589), 5e50b5e8a58546bdb6509bba0687350b (127.31.32.3:36469)
I20260810 04:07:08.695530 8051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "354755d4f25a4771bdc3a321c4defef8" candidate_uuid: "1d4dabac174c4e2bbc75a10c2fa03a4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e50b5e8a58546bdb6509bba0687350b" is_pre_election: true
I20260810 04:07:08.695530 7899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "354755d4f25a4771bdc3a321c4defef8" candidate_uuid: "1d4dabac174c4e2bbc75a10c2fa03a4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f000d641e9094650bbd16cf6b7a35069" is_pre_election: true
I20260810 04:07:08.695660 8051 raft_consensus.cc:2454] T 354755d4f25a4771bdc3a321c4defef8 P 5e50b5e8a58546bdb6509bba0687350b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d4dabac174c4e2bbc75a10c2fa03a4c in term 0.
I20260810 04:07:08.695660 7899 raft_consensus.cc:2454] T 354755d4f25a4771bdc3a321c4defef8 P f000d641e9094650bbd16cf6b7a35069 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d4dabac174c4e2bbc75a10c2fa03a4c in term 0.
I20260810 04:07:08.695808 7671 leader_election.cc:302] T 354755d4f25a4771bdc3a321c4defef8 P 1d4dabac174c4e2bbc75a10c2fa03a4c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d4dabac174c4e2bbc75a10c2fa03a4c, 5e50b5e8a58546bdb6509bba0687350b; no voters:
I20260810 04:07:08.695923 8213 raft_consensus.cc:2781] T 354755d4f25a4771bdc3a321c4defef8 P 1d4dabac174c4e2bbc75a10c2fa03a4c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 04:07:08.695957 8213 raft_consensus.cc:482] T 354755d4f25a4771bdc3a321c4defef8 P 1d4dabac174c4e2bbc75a10c2fa03a4c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 04:07:08.695984 8213 raft_consensus.cc:3037] T 354755d4f25a4771bdc3a321c4defef8 P 1d4dabac174c4e2bbc75a10c2fa03a4c [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:07:08.696767 8213 raft_consensus.cc:504] T 354755d4f25a4771bdc3a321c4defef8 P 1d4dabac174c4e2bbc75a10c2fa03a4c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d4dabac174c4e2bbc75a10c2fa03a4c" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 46371 } } peers { permanent_uuid: "f000d641e9094650bbd16cf6b7a35069" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 44589 } } peers { permanent_uuid: "5e50b5e8a58546bdb6509bba0687350b" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 36469 } }
I20260810 04:07:08.696916 8213 leader_election.cc:288] T 354755d4f25a4771bdc3a321c4defef8 P 1d4dabac174c4e2bbc75a10c2fa03a4c [CANDIDATE]: Term 1 election: Requested vote from peers f000d641e9094650bbd16cf6b7a35069 (127.31.32.2:44589), 5e50b5e8a58546bdb6509bba0687350b (127.31.32.3:36469)
I20260810 04:07:08.697073 8051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "354755d4f25a4771bdc3a321c4defef8" candidate_uuid: "1d4dabac174c4e2bbc75a10c2fa03a4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e50b5e8a58546bdb6509bba0687350b"
I20260810 04:07:08.697073 7899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "354755d4f25a4771bdc3a321c4defef8" candidate_uuid: "1d4dabac174c4e2bbc75a10c2fa03a4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f000d641e9094650bbd16cf6b7a35069"
I20260810 04:07:08.697151 8051 raft_consensus.cc:3037] T 354755d4f25a4771bdc3a321c4defef8 P 5e50b5e8a58546bdb6509bba0687350b [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:07:08.697177 7899 raft_consensus.cc:3037] T 354755d4f25a4771bdc3a321c4defef8 P f000d641e9094650bbd16cf6b7a35069 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:07:08.697767 8051 raft_consensus.cc:2454] T 354755d4f25a4771bdc3a321c4defef8 P 5e50b5e8a58546bdb6509bba0687350b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d4dabac174c4e2bbc75a10c2fa03a4c in term 1.
I20260810 04:07:08.697897 7899 raft_consensus.cc:2454] T 354755d4f25a4771bdc3a321c4defef8 P f000d641e9094650bbd16cf6b7a35069 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d4dabac174c4e2bbc75a10c2fa03a4c in term 1.
I20260810 04:07:08.697935 7671 leader_election.cc:302] T 354755d4f25a4771bdc3a321c4defef8 P 1d4dabac174c4e2bbc75a10c2fa03a4c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d4dabac174c4e2bbc75a10c2fa03a4c, 5e50b5e8a58546bdb6509bba0687350b; no voters:
I20260810 04:07:08.698048 8213 raft_consensus.cc:2781] T 354755d4f25a4771bdc3a321c4defef8 P 1d4dabac174c4e2bbc75a10c2fa03a4c [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:07:08.698115 8213 raft_consensus.cc:686] T 354755d4f25a4771bdc3a321c4defef8 P 1d4dabac174c4e2bbc75a10c2fa03a4c [term 1 LEADER]: Becoming Leader. State: Replica: 1d4dabac174c4e2bbc75a10c2fa03a4c, State: Running, Role: LEADER
I20260810 04:07:08.698173 8213 consensus_queue.cc:237] T 354755d4f25a4771bdc3a321c4defef8 P 1d4dabac174c4e2bbc75a10c2fa03a4c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d4dabac174c4e2bbc75a10c2fa03a4c" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 46371 } } peers { permanent_uuid: "f000d641e9094650bbd16cf6b7a35069" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 44589 } } peers { permanent_uuid: "5e50b5e8a58546bdb6509bba0687350b" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 36469 } }
I20260810 04:07:08.698800 7566 catalog_manager.cc:5495] T 354755d4f25a4771bdc3a321c4defef8 P 1d4dabac174c4e2bbc75a10c2fa03a4c reported cstate change: term changed from 0 to 1, leader changed from <none> to 1d4dabac174c4e2bbc75a10c2fa03a4c (127.31.32.1). New cstate: current_term: 1 leader_uuid: "1d4dabac174c4e2bbc75a10c2fa03a4c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d4dabac174c4e2bbc75a10c2fa03a4c" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 46371 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f000d641e9094650bbd16cf6b7a35069" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 44589 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e50b5e8a58546bdb6509bba0687350b" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 36469 } health_report { overall_health: UNKNOWN } } }
I20260810 04:07:08.744546 8129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:08.742230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.1:48467 (local address 127.31.32.3:36469)
0810 04:07:08.742273 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0810 04:07:08.742276 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:07:08.742388 (+ 112us) server_negotiation.cc:407] Connection header received
0810 04:07:08.742415 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:07:08.742415 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:07:08.742444 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:07:08.742508 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 04:07:08.742614 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:08.743084 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:08.743836 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:08.744021 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:08.744115 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:08.744317 (+ 202us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 04:07:08.744361 (+ 44us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 04:07:08.744415 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0810 04:07:08.744469 (+ 54us) server_negotiation.cc:299] Negotiation successful
0810 04:07:08.744490 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260810 04:07:08.744577 8226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:08.742111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.3:36469 (local address 127.31.32.1:48467)
0810 04:07:08.742303 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0810 04:07:08.742313 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 04:07:08.742381 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:07:08.742524 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:07:08.742524 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:07:08.742526 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 04:07:08.742573 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:08.742574 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:08.743146 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:08.743146 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:08.743727 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:08.743729 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:08.744148 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:08.744149 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:08.744204 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:08.744232 (+ 28us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 04:07:08.744429 (+ 197us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 04:07:08.744430 (+ 1us) client_negotiation.cc:769] Sending connection context
0810 04:07:08.744466 (+ 36us) client_negotiation.cc:240] Negotiation successful
0810 04:07:08.744502 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":116,"threads_started":1}
I20260810 04:07:08.746618 7899 raft_consensus.cc:1261] T 354755d4f25a4771bdc3a321c4defef8 P f000d641e9094650bbd16cf6b7a35069 [term 1 FOLLOWER]: Refusing update from remote peer 1d4dabac174c4e2bbc75a10c2fa03a4c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:07:08.746634 8051 raft_consensus.cc:1261] T 354755d4f25a4771bdc3a321c4defef8 P 5e50b5e8a58546bdb6509bba0687350b [term 1 FOLLOWER]: Refusing update from remote peer 1d4dabac174c4e2bbc75a10c2fa03a4c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:07:08.746925 8141 consensus_queue.cc:1035] T 354755d4f25a4771bdc3a321c4defef8 P 1d4dabac174c4e2bbc75a10c2fa03a4c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f000d641e9094650bbd16cf6b7a35069" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 44589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:07:08.747051 8213 consensus_queue.cc:1035] T 354755d4f25a4771bdc3a321c4defef8 P 1d4dabac174c4e2bbc75a10c2fa03a4c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e50b5e8a58546bdb6509bba0687350b" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 36469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:07:08.754164 8131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:08.751919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.3:39577 (local address 127.31.32.1:46371)
0810 04:07:08.751988 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0810 04:07:08.751990 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:07:08.751998 (+ 8us) server_negotiation.cc:407] Connection header received
0810 04:07:08.752017 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:07:08.752018 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:07:08.752027 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:07:08.752082 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 04:07:08.752217 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:08.752647 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:08.753389 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:08.753681 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:08.753750 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:08.753920 (+ 170us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 04:07:08.753955 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 04:07:08.754003 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0810 04:07:08.754095 (+ 92us) server_negotiation.cc:299] Negotiation successful
0810 04:07:08.754116 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260810 04:07:08.754176 8191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:08.751693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.1:46371 (local address 127.31.32.3:39577)
0810 04:07:08.751764 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0810 04:07:08.751776 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 04:07:08.751822 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:07:08.752106 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:07:08.752107 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:07:08.752108 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 04:07:08.752162 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:08.752163 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:08.752708 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:08.752709 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:08.753287 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:08.753289 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:08.753766 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:08.753767 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:08.753827 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:08.753834 (+ 7us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 04:07:08.754032 (+ 198us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 04:07:08.754035 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 04:07:08.754075 (+ 40us) client_negotiation.cc:240] Negotiation successful
0810 04:07:08.754126 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260810 04:07:08.760005 7568 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44974:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 04:07:08.760109 7568 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44974:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 04:07:08.760769 7568 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2c147e41e15f4984b013b451708c65b3: Sending DeleteTablet for 3 replicas of tablet 354755d4f25a4771bdc3a321c4defef8
I20260810 04:07:08.761008 7720 tablet_service.cc:1514] Processing DeleteTablet for tablet 354755d4f25a4771bdc3a321c4defef8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:07:08 UTC) from {username='oryx', principal='oryx/127.31.32.62@KRBTEST.COM'} at 127.0.0.1:35138
I20260810 04:07:08.761019 8028 tablet_service.cc:1514] Processing DeleteTablet for tablet 354755d4f25a4771bdc3a321c4defef8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:07:08 UTC) from {username='oryx', principal='oryx/127.31.32.62@KRBTEST.COM'} at 127.0.0.1:36476
I20260810 04:07:08.761121 8160 tablet_replica.cc:330] T 354755d4f25a4771bdc3a321c4defef8 P 5e50b5e8a58546bdb6509bba0687350b: stopping tablet replica
I20260810 04:07:08.761087 7875 tablet_service.cc:1514] Processing DeleteTablet for tablet 354755d4f25a4771bdc3a321c4defef8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:07:08 UTC) from {username='oryx', principal='oryx/127.31.32.62@KRBTEST.COM'} at 127.0.0.1:54646
I20260810 04:07:08.761168 8160 raft_consensus.cc:2229] T 354755d4f25a4771bdc3a321c4defef8 P 5e50b5e8a58546bdb6509bba0687350b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 04:07:08.761201 8160 raft_consensus.cc:2258] T 354755d4f25a4771bdc3a321c4defef8 P 5e50b5e8a58546bdb6509bba0687350b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:07:08.761209 8159 tablet_replica.cc:330] T 354755d4f25a4771bdc3a321c4defef8 P 1d4dabac174c4e2bbc75a10c2fa03a4c: stopping tablet replica
I20260810 04:07:08.761277 8159 raft_consensus.cc:2229] T 354755d4f25a4771bdc3a321c4defef8 P 1d4dabac174c4e2bbc75a10c2fa03a4c [term 1 LEADER]: Raft consensus shutting down.
I20260810 04:07:08.761345 8159 raft_consensus.cc:2258] T 354755d4f25a4771bdc3a321c4defef8 P 1d4dabac174c4e2bbc75a10c2fa03a4c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:07:08.761430 8161 tablet_replica.cc:330] T 354755d4f25a4771bdc3a321c4defef8 P f000d641e9094650bbd16cf6b7a35069: stopping tablet replica
I20260810 04:07:08.761497 8161 raft_consensus.cc:2229] T 354755d4f25a4771bdc3a321c4defef8 P f000d641e9094650bbd16cf6b7a35069 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 04:07:08.761543 8161 raft_consensus.cc:2258] T 354755d4f25a4771bdc3a321c4defef8 P f000d641e9094650bbd16cf6b7a35069 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:07:08.761607 8159 ts_tablet_manager.cc:1905] T 354755d4f25a4771bdc3a321c4defef8 P 1d4dabac174c4e2bbc75a10c2fa03a4c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:07:08.761722 31872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskws89Io/build/release/bin/kudu with pid 7650
I20260810 04:07:08.763809 8160 ts_tablet_manager.cc:1905] T 354755d4f25a4771bdc3a321c4defef8 P 5e50b5e8a58546bdb6509bba0687350b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:07:08.765475 8161 ts_tablet_manager.cc:1905] T 354755d4f25a4771bdc3a321c4defef8 P f000d641e9094650bbd16cf6b7a35069: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:07:08.765657 8160 ts_tablet_manager.cc:1918] T 354755d4f25a4771bdc3a321c4defef8 P 5e50b5e8a58546bdb6509bba0687350b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 04:07:08.765722 8160 log.cc:1196] T 354755d4f25a4771bdc3a321c4defef8 P 5e50b5e8a58546bdb6509bba0687350b: Deleting WAL directory at /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786334794676589-31872-0/minicluster-data/ts-2/wal/wals/354755d4f25a4771bdc3a321c4defef8
I20260810 04:07:08.765930 8160 ts_tablet_manager.cc:1939] T 354755d4f25a4771bdc3a321c4defef8 P 5e50b5e8a58546bdb6509bba0687350b: Deleting consensus metadata
I20260810 04:07:08.766347 7554 catalog_manager.cc:4841] TS 5e50b5e8a58546bdb6509bba0687350b (127.31.32.3:36469): tablet 354755d4f25a4771bdc3a321c4defef8 (table test-table [id=02bfb23f6c6a470abbc66d00e7c57963]) successfully deleted
I20260810 04:07:08.766878 8161 ts_tablet_manager.cc:1918] T 354755d4f25a4771bdc3a321c4defef8 P f000d641e9094650bbd16cf6b7a35069: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 04:07:08.766960 8161 log.cc:1196] T 354755d4f25a4771bdc3a321c4defef8 P f000d641e9094650bbd16cf6b7a35069: Deleting WAL directory at /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786334794676589-31872-0/minicluster-data/ts-1/wal/wals/354755d4f25a4771bdc3a321c4defef8
I20260810 04:07:08.767200 8161 ts_tablet_manager.cc:1939] T 354755d4f25a4771bdc3a321c4defef8 P f000d641e9094650bbd16cf6b7a35069: Deleting consensus metadata
I20260810 04:07:08.767555 7553 catalog_manager.cc:4841] TS f000d641e9094650bbd16cf6b7a35069 (127.31.32.2:44589): tablet 354755d4f25a4771bdc3a321c4defef8 (table test-table [id=02bfb23f6c6a470abbc66d00e7c57963]) successfully deleted
W20260810 04:07:08.770973 7553 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.32.1:46371 (error 108)
I20260810 04:07:08.771173 31872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskws89Io/build/release/bin/kudu with pid 7804
I20260810 04:07:08.772806 8231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:08.772616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.1:46371 (local address 127.0.0.1:35154)
0810 04:07:08.772725 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0810 04:07:08.772764 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.1:46371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":39,"threads_started":1}
W20260810 04:07:08.772895 7553 catalog_manager.cc:4584] TS 1d4dabac174c4e2bbc75a10c2fa03a4c (127.31.32.1:46371): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 354755d4f25a4771bdc3a321c4defef8: Network error: Client connection negotiation failed: client connection to 127.31.32.1:46371: connect: Connection refused (error 111)
I20260810 04:07:08.779006 31872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskws89Io/build/release/bin/kudu with pid 7958
I20260810 04:07:08.786909 31872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskws89Io/build/release/bin/kudu with pid 7535
2026-08-10T04:07:08Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (2772 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786334794676589-31872-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 04:07:08 dist-test-slave-8w9v krb5kdc[8236](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 04:07:08 dist-test-slave-8w9v krb5kdc[8236](info): set up 2 sockets
Aug 10 04:07:08 dist-test-slave-8w9v krb5kdc[8236](info): commencing operation
krb5kdc: starting...
W20260810 04:07:09.905292 31872 mini_kdc.cc:121] Time spent starting KDC: real 1.095s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786334794676589-31872-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786334794676589-31872-0/krb5kdc/test-user.keytab.
Aug 10 04:07:09 dist-test-slave-8w9v krb5kdc[8236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334829, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T04:07:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T04:07:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.32.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.32.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.32.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.62.keytab.
Entry for principal kudu/127.31.32.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.62.keytab.
I20260810 04:07:10.024981 31872 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskws89Io/build/release/bin/kudu
/tmp/dist-test-taskws89Io/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786334794676589-31872-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786334794676589-31872-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786334794676589-31872-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786334794676589-31872-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.32.62:34223
--webserver_interface=127.31.32.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.62.keytab
--principal=kudu/127.31.32.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.32.20:46775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.32.62:34223
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786334794676589-31872-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786334794676589-31872-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786334794676589-31872-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:07:10.108222 8251 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:07:10.108417 8251 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:07:10.108440 8251 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:07:10.109851 8251 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:07:10.109894 8251 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:07:10.109954 8251 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:07:10.109973 8251 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:07:10.109997 8251 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:07:10.111531 8251 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.32.20:46775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786334794676589-31872-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786334794676589-31872-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.32.62:34223
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.32.62:34223
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786334794676589-31872-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.32.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8251
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786334794676589-31872-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:07:10.111792 8251 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:07:10.112036 8251 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:07:10.116958 8256 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:07:10.117120 8251 server_base.cc:1034] running on GCE node
W20260810 04:07:10.116938 8259 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:07:10.116919 8257 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:07:10.117537 8251 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:07:10.117781 8251 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:07:10.118929 8251 hybrid_clock.cc:648] HybridClock initialized: now 1786334830118892 us; error 53 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-08-10T04:07:10Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1332 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 04:07:10 dist-test-slave-8w9v krb5kdc[8265](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 04:07:10 dist-test-slave-8w9v krb5kdc[8265](info): set up 2 sockets
Aug 10 04:07:10 dist-test-slave-8w9v krb5kdc[8265](info): commencing operation
krb5kdc: starting...
W20260810 04:07:11.192503 31872 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/krb5kdc/test-user.keytab.
Aug 10 04:07:11 dist-test-slave-8w9v krb5kdc[8265](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334831, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T04:07:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T04:07:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.32.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.32.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.32.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.62.keytab.
Entry for principal kudu/127.31.32.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.62.keytab.
I20260810 04:07:11.309021 31872 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskws89Io/build/release/bin/kudu
/tmp/dist-test-taskws89Io/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.32.62:43401
--webserver_interface=127.31.32.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.62.keytab
--principal=kudu/127.31.32.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.32.20:41699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.32.62:43401
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:07:11.389562 8279 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:07:11.389766 8279 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:07:11.389791 8279 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:07:11.390972 8279 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:07:11.391021 8279 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:07:11.391036 8279 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:07:11.391050 8279 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:07:11.391108 8279 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:07:11.392552 8279 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.32.20:41699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.32.62:43401
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.62.keytab
--principal=kudu/127.31.32.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.32.62:43401
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.32.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8279
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:07:11.392819 8279 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:07:11.393050 8279 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:07:11.397804 8287 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:07:11.397826 8284 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:07:11.397886 8279 server_base.cc:1034] running on GCE node
W20260810 04:07:11.397837 8285 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:07:11.398357 8279 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:07:11.398650 8279 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:07:11.399823 8279 hybrid_clock.cc:648] HybridClock initialized: now 1786334831399816 us; error 35 us; skew 500 ppm
Aug 10 04:07:11 dist-test-slave-8w9v krb5kdc[8265](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334831, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:07:11.406673 8279 init.cc:377] Logged in from keytab as kudu/127.31.32.62@KRBTEST.COM (short username kudu)
I20260810 04:07:11.407099 8279 webserver.cc:460] Webserver started at http://127.31.32.62:34481/ using document root <none> and password file <none>
I20260810 04:07:11.407351 8279 fs_manager.cc:359] Metadata directory not provided
I20260810 04:07:11.407420 8279 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:07:11.407567 8279 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:07:11.408702 8279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/minicluster-data/master-0/data/instance:
uuid: "7e1ed946ff53405b95e9c249a76361e8"
format_stamp: "Formatted at 2026-08-10 04:07:11 on dist-test-slave-8w9v"
server_key: "d9464e398b80b59952a6589d8eaa715a"
server_key_iv: "297178dafee0b7f5873da9275babae7c"
server_key_version: "encryptionkey@0"
I20260810 04:07:11.409052 8279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/minicluster-data/master-0/wal/instance:
uuid: "7e1ed946ff53405b95e9c249a76361e8"
format_stamp: "Formatted at 2026-08-10 04:07:11 on dist-test-slave-8w9v"
server_key: "d9464e398b80b59952a6589d8eaa715a"
server_key_iv: "297178dafee0b7f5873da9275babae7c"
server_key_version: "encryptionkey@0"
I20260810 04:07:11.410359 8279 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 04:07:11.411039 8295 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:07:11.411213 8279 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 04:07:11.411317 8279 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/minicluster-data/master-0/data,/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/minicluster-data/master-0/wal
uuid: "7e1ed946ff53405b95e9c249a76361e8"
format_stamp: "Formatted at 2026-08-10 04:07:11 on dist-test-slave-8w9v"
server_key: "d9464e398b80b59952a6589d8eaa715a"
server_key_iv: "297178dafee0b7f5873da9275babae7c"
server_key_version: "encryptionkey@0"
I20260810 04:07:11.411404 8279 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:07:11.418362 8279 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:07:11.420231 8279 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:07:11.420393 8279 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:07:11.424201 8279 rpc_server.cc:307] RPC server started. Bound to: 127.31.32.62:43401
I20260810 04:07:11.424233 8352 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.32.62:43401 every 8 connection(s)
I20260810 04:07:11.424554 8279 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/minicluster-data/master-0/data/info.pb
I20260810 04:07:11.425426 8353 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:07:11.427573 8353 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e1ed946ff53405b95e9c249a76361e8: Bootstrap starting.
I20260810 04:07:11.428207 8353 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7e1ed946ff53405b95e9c249a76361e8: Neither blocks nor log segments found. Creating new log.
I20260810 04:07:11.428500 8353 log.cc:824] T 00000000000000000000000000000000 P 7e1ed946ff53405b95e9c249a76361e8: Log is configured to *not* fsync() on all Append() calls
I20260810 04:07:11.429183 8353 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e1ed946ff53405b95e9c249a76361e8: No bootstrap required, opened a new log
I20260810 04:07:11.430789 8353 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7e1ed946ff53405b95e9c249a76361e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e1ed946ff53405b95e9c249a76361e8" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 43401 } }
I20260810 04:07:11.430930 8353 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7e1ed946ff53405b95e9c249a76361e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:07:11.430953 8353 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7e1ed946ff53405b95e9c249a76361e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e1ed946ff53405b95e9c249a76361e8, State: Initialized, Role: FOLLOWER
I20260810 04:07:11.431113 8353 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7e1ed946ff53405b95e9c249a76361e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e1ed946ff53405b95e9c249a76361e8" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 43401 } }
I20260810 04:07:11.431174 8353 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7e1ed946ff53405b95e9c249a76361e8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 04:07:11.431195 8353 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7e1ed946ff53405b95e9c249a76361e8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 04:07:11.431259 8353 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7e1ed946ff53405b95e9c249a76361e8 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:07:11.431744 8353 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7e1ed946ff53405b95e9c249a76361e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e1ed946ff53405b95e9c249a76361e8" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 43401 } }
I20260810 04:07:11.431859 8353 leader_election.cc:302] T 00000000000000000000000000000000 P 7e1ed946ff53405b95e9c249a76361e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7e1ed946ff53405b95e9c249a76361e8; no voters:
I20260810 04:07:11.432034 8353 leader_election.cc:288] T 00000000000000000000000000000000 P 7e1ed946ff53405b95e9c249a76361e8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 04:07:11.432101 8357 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7e1ed946ff53405b95e9c249a76361e8 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:07:11.432312 8353 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7e1ed946ff53405b95e9c249a76361e8 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:07:11.432564 8357 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7e1ed946ff53405b95e9c249a76361e8 [term 1 LEADER]: Becoming Leader. State: Replica: 7e1ed946ff53405b95e9c249a76361e8, State: Running, Role: LEADER
I20260810 04:07:11.432726 8357 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7e1ed946ff53405b95e9c249a76361e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e1ed946ff53405b95e9c249a76361e8" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 43401 } }
I20260810 04:07:11.433156 8358 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e1ed946ff53405b95e9c249a76361e8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7e1ed946ff53405b95e9c249a76361e8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e1ed946ff53405b95e9c249a76361e8" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 43401 } } }
I20260810 04:07:11.433168 8359 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e1ed946ff53405b95e9c249a76361e8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7e1ed946ff53405b95e9c249a76361e8. Latest consensus state: current_term: 1 leader_uuid: "7e1ed946ff53405b95e9c249a76361e8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e1ed946ff53405b95e9c249a76361e8" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 43401 } } }
I20260810 04:07:11.433256 8358 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e1ed946ff53405b95e9c249a76361e8 [sys.catalog]: This master's current role is: LEADER
I20260810 04:07:11.433286 8359 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e1ed946ff53405b95e9c249a76361e8 [sys.catalog]: This master's current role is: LEADER
I20260810 04:07:11.433452 31872 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskws89Io/build/release/bin/kudu as pid 8279
I20260810 04:07:11.433463 8366 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 04:07:11.433559 31872 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/minicluster-data/master-0/wal/instance
I20260810 04:07:11.433773 31872 external_mini_cluster.cc:1428] Setting key f36c6413a1aa9fb3788c72b7a4805b70
I20260810 04:07:11.433947 8366 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 04:07:11.435390 8366 catalog_manager.cc:1347] Generated new cluster ID: 1e88a24b93054b5a9d1da3c33f382054
I20260810 04:07:11.435442 8366 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 10 04:07:11 dist-test-slave-8w9v krb5kdc[8265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334831, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.32.62@KRBTEST.COM
I20260810 04:07:11.453142 8378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:11.434405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57664 (local address 127.31.32.62:43401)
0810 04:07:11.442447 (+ 8042us) server_negotiation.cc:206] Beginning negotiation
0810 04:07:11.442456 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 04:07:11.442488 (+ 32us) server_negotiation.cc:407] Connection header received
0810 04:07:11.442594 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:07:11.442601 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:07:11.442658 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:07:11.442781 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:07:11.443577 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:11.444988 (+ 1411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:11.446043 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:11.446615 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:11.446728 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:11.449416 (+ 2688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:07:11.449420 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:07:11.449421 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:07:11.449432 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:07:11.451319 (+ 1887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:07:11.451699 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:07:11.451701 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:07:11.451702 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:07:11.451820 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:07:11.452023 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:07:11.452023 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:07:11.452024 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:07:11.452335 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:07:11.452506 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0810 04:07:11.452592 (+ 86us) server_negotiation.cc:299] Negotiation successful
0810 04:07:11.452725 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7994,"thread_start_us":72,"threads_started":1}
I20260810 04:07:11.453480 8366 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 04:07:11.454053 8366 catalog_manager.cc:1504] Loading token signing keys...
I20260810 04:07:11.471901 8366 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7e1ed946ff53405b95e9c249a76361e8: Generated new TSK 0
I20260810 04:07:11.472090 8366 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.32.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.32.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.32.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.1.keytab.
Entry for principal kudu/127.31.32.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.1.keytab.
I20260810 04:07:11.507133 31872 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskws89Io/build/release/bin/kudu
/tmp/dist-test-taskws89Io/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.32.1:0
--local_ip_for_outbound_sockets=127.31.32.1
--webserver_interface=127.31.32.1
--webserver_port=0
--tserver_master_addrs=127.31.32.62:43401
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.1.keytab
--principal=kudu/127.31.32.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.32.20:41699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:07:11.587265 8382 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:07:11.587457 8382 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:07:11.587481 8382 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:07:11.588614 8382 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:07:11.588660 8382 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:07:11.588682 8382 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.32.1
I20260810 04:07:11.590021 8382 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.32.20:41699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.1.keytab
--principal=kudu/127.31.32.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.32.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.32.1
--webserver_port=0
--tserver_master_addrs=127.31.32.62:43401
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8382
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.32.1
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:07:11.590266 8382 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:07:11.590495 8382 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:07:11.595323 8390 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:07:11.595301 8388 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:07:11.595314 8387 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:07:11.595346 8382 server_base.cc:1034] running on GCE node
I20260810 04:07:11.595685 8382 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:07:11.595919 8382 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:07:11.597108 8382 hybrid_clock.cc:648] HybridClock initialized: now 1786334831597105 us; error 54 us; skew 500 ppm
Aug 10 04:07:11 dist-test-slave-8w9v krb5kdc[8265](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334831, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:07:11.599673 8382 init.cc:377] Logged in from keytab as kudu/127.31.32.1@KRBTEST.COM (short username kudu)
I20260810 04:07:11.600191 8382 webserver.cc:460] Webserver started at http://127.31.32.1:38341/ using document root <none> and password file <none>
I20260810 04:07:11.600435 8382 fs_manager.cc:359] Metadata directory not provided
I20260810 04:07:11.600504 8382 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:07:11.600662 8382 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:07:11.601717 8382 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/minicluster-data/ts-0/data/instance:
uuid: "782397279dd24fbe8ef43831f3dcb8e5"
format_stamp: "Formatted at 2026-08-10 04:07:11 on dist-test-slave-8w9v"
server_key: "1b6df4d1ecce7874ce0f04f2ce03729c"
server_key_iv: "dd7224328f3497ad788f1a66a4ac340d"
server_key_version: "encryptionkey@0"
I20260810 04:07:11.602089 8382 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/minicluster-data/ts-0/wal/instance:
uuid: "782397279dd24fbe8ef43831f3dcb8e5"
format_stamp: "Formatted at 2026-08-10 04:07:11 on dist-test-slave-8w9v"
server_key: "1b6df4d1ecce7874ce0f04f2ce03729c"
server_key_iv: "dd7224328f3497ad788f1a66a4ac340d"
server_key_version: "encryptionkey@0"
I20260810 04:07:11.603405 8382 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 04:07:11.604074 8400 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:07:11.604238 8382 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 04:07:11.604334 8382 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/minicluster-data/ts-0/data,/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/minicluster-data/ts-0/wal
uuid: "782397279dd24fbe8ef43831f3dcb8e5"
format_stamp: "Formatted at 2026-08-10 04:07:11 on dist-test-slave-8w9v"
server_key: "1b6df4d1ecce7874ce0f04f2ce03729c"
server_key_iv: "dd7224328f3497ad788f1a66a4ac340d"
server_key_version: "encryptionkey@0"
I20260810 04:07:11.604385 8382 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:07:11.618132 8382 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:07:11.620553 8382 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:07:11.620657 8382 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:07:11.620847 8382 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:07:11.621167 8382 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:07:11.621228 8382 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:07:11.621269 8382 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:07:11.621304 8382 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:07:11.626748 8382 rpc_server.cc:307] RPC server started. Bound to: 127.31.32.1:40891
I20260810 04:07:11.626816 8525 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.32.1:40891 every 8 connection(s)
I20260810 04:07:11.627133 8382 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/minicluster-data/ts-0/data/info.pb
I20260810 04:07:11.632095 31872 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskws89Io/build/release/bin/kudu as pid 8382
I20260810 04:07:11.632236 31872 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/minicluster-data/ts-0/wal/instance
I20260810 04:07:11.632476 31872 external_mini_cluster.cc:1428] Setting key 3147defbc6e4525ee4252ed8e42958b6
Aug 10 04:07:11 dist-test-slave-8w9v krb5kdc[8265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334831, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.1@KRBTEST.COM for kudu/127.31.32.62@KRBTEST.COM
I20260810 04:07:11.636806 8378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:11.627830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.1:42655 (local address 127.31.32.62:43401)
0810 04:07:11.627913 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0810 04:07:11.627915 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:07:11.628084 (+ 169us) server_negotiation.cc:407] Connection header received
0810 04:07:11.628630 (+ 546us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:07:11.628631 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:07:11.628645 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:07:11.628694 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:07:11.629581 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:11.630085 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:11.631072 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:11.631285 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:11.631374 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:11.633763 (+ 2389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:07:11.633764 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:07:11.633765 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:07:11.633771 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:07:11.635442 (+ 1671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:07:11.635816 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:07:11.635817 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:07:11.635818 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:07:11.635885 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:07:11.636167 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:07:11.636168 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:07:11.636169 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:07:11.636298 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:07:11.636353 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0810 04:07:11.636675 (+ 322us) server_negotiation.cc:299] Negotiation successful
0810 04:07:11.636742 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260810 04:07:11.637073 8528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:11.627785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:43401 (local address 127.31.32.1:42655)
0810 04:07:11.627959 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0810 04:07:11.628003 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 04:07:11.628517 (+ 514us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:07:11.628757 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:07:11.628761 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:07:11.629292 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:07:11.629474 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:11.629479 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:11.630181 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:11.630183 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:11.630983 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:11.630985 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:11.631388 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:11.631389 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:11.631519 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:11.631937 (+ 418us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:07:11.631947 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:07:11.633651 (+ 1704us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:07:11.635523 (+ 1872us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:07:11.635529 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:07:11.635531 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:07:11.635742 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:07:11.635960 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:07:11.635961 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:07:11.635961 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:07:11.636060 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:07:11.636367 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:07:11.636374 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:07:11.636531 (+ 157us) client_negotiation.cc:769] Sending connection context
0810 04:07:11.636641 (+ 110us) client_negotiation.cc:240] Negotiation successful
0810 04:07:11.636797 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":88,"threads_started":1}
I20260810 04:07:11.637708 8526 heartbeater.cc:347] Connected to a master server at 127.31.32.62:43401
I20260810 04:07:11.637825 8526 heartbeater.cc:464] Registering TS with master...
I20260810 04:07:11.638015 8526 heartbeater.cc:511] Master 127.31.32.62:43401 requested a full tablet report, sending...
I20260810 04:07:11.638486 8314 ts_manager.cc:194] Registered new tserver with Master: 782397279dd24fbe8ef43831f3dcb8e5 (127.31.32.1:40891)
I20260810 04:07:11.639058 8314 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.32.1@KRBTEST.COM'} at 127.31.32.1:42655
WARNING: no policy specified for kudu/127.31.32.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.32.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.32.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.2.keytab.
Entry for principal kudu/127.31.32.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.2.keytab.
I20260810 04:07:11.657021 31872 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskws89Io/build/release/bin/kudu
/tmp/dist-test-taskws89Io/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.32.2:0
--local_ip_for_outbound_sockets=127.31.32.2
--webserver_interface=127.31.32.2
--webserver_port=0
--tserver_master_addrs=127.31.32.62:43401
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.2.keytab
--principal=kudu/127.31.32.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.32.20:41699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:07:11.735734 8531 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:07:11.735961 8531 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:07:11.736014 8531 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:07:11.737236 8531 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:07:11.737303 8531 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:07:11.737346 8531 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.32.2
I20260810 04:07:11.738771 8531 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.32.20:41699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.2.keytab
--principal=kudu/127.31.32.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.32.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.32.2
--webserver_port=0
--tserver_master_addrs=127.31.32.62:43401
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8531
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.32.2
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:07:11.739042 8531 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:07:11.739264 8531 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:07:11.744136 8538 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:07:11.744149 8540 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:07:11.744350 8531 server_base.cc:1034] running on GCE node
W20260810 04:07:11.744149 8537 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:07:11.744689 8531 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:07:11.744952 8531 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:07:11.748170 8531 hybrid_clock.cc:648] HybridClock initialized: now 1786334831748145 us; error 34 us; skew 500 ppm
Aug 10 04:07:11 dist-test-slave-8w9v krb5kdc[8265](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334831, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:07:11.751590 8531 init.cc:377] Logged in from keytab as kudu/127.31.32.2@KRBTEST.COM (short username kudu)
I20260810 04:07:11.752084 8531 webserver.cc:460] Webserver started at http://127.31.32.2:41135/ using document root <none> and password file <none>
I20260810 04:07:11.752347 8531 fs_manager.cc:359] Metadata directory not provided
I20260810 04:07:11.752386 8531 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:07:11.752486 8531 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:07:11.753454 8531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/minicluster-data/ts-1/data/instance:
uuid: "ecc296d748d741069f4debb361e79664"
format_stamp: "Formatted at 2026-08-10 04:07:11 on dist-test-slave-8w9v"
server_key: "4e0e42e15a87e9c118b7d1577120970d"
server_key_iv: "68cc980deb195df05187440780488fa6"
server_key_version: "encryptionkey@0"
I20260810 04:07:11.753731 8531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/minicluster-data/ts-1/wal/instance:
uuid: "ecc296d748d741069f4debb361e79664"
format_stamp: "Formatted at 2026-08-10 04:07:11 on dist-test-slave-8w9v"
server_key: "4e0e42e15a87e9c118b7d1577120970d"
server_key_iv: "68cc980deb195df05187440780488fa6"
server_key_version: "encryptionkey@0"
I20260810 04:07:11.754999 8531 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 04:07:11.755671 8547 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:07:11.755848 8531 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 04:07:11.755945 8531 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/minicluster-data/ts-1/data,/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/minicluster-data/ts-1/wal
uuid: "ecc296d748d741069f4debb361e79664"
format_stamp: "Formatted at 2026-08-10 04:07:11 on dist-test-slave-8w9v"
server_key: "4e0e42e15a87e9c118b7d1577120970d"
server_key_iv: "68cc980deb195df05187440780488fa6"
server_key_version: "encryptionkey@0"
I20260810 04:07:11.756044 8531 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:07:11.765571 8531 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:07:11.767396 8531 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:07:11.767549 8531 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:07:11.767760 8531 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:07:11.768064 8531 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:07:11.768118 8531 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:07:11.768200 8531 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:07:11.768234 8531 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:07:11.774000 8531 rpc_server.cc:307] RPC server started. Bound to: 127.31.32.2:33853
I20260810 04:07:11.774071 8677 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.32.2:33853 every 8 connection(s)
I20260810 04:07:11.774355 8531 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/minicluster-data/ts-1/data/info.pb
Aug 10 04:07:11 dist-test-slave-8w9v krb5kdc[8265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334831, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.2@KRBTEST.COM for kudu/127.31.32.62@KRBTEST.COM
I20260810 04:07:11.781250 31872 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskws89Io/build/release/bin/kudu as pid 8531
I20260810 04:07:11.781347 31872 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/minicluster-data/ts-1/wal/instance
I20260810 04:07:11.781577 31872 external_mini_cluster.cc:1428] Setting key 642468cb70adc3eb329dfb7d5b0abd27
I20260810 04:07:11.783195 8378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:11.775041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.2:49387 (local address 127.31.32.62:43401)
0810 04:07:11.775126 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0810 04:07:11.775128 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:07:11.775250 (+ 122us) server_negotiation.cc:407] Connection header received
0810 04:07:11.775692 (+ 442us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:07:11.775694 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:07:11.775708 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:07:11.775759 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:07:11.776658 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:11.777131 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:11.778047 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:11.778223 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:11.778272 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:11.780247 (+ 1975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:07:11.780249 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:07:11.780249 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:07:11.780274 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:07:11.781747 (+ 1473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:07:11.782215 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:07:11.782217 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:07:11.782217 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:07:11.782287 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:07:11.782586 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:07:11.782586 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:07:11.782587 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:07:11.782690 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:07:11.782740 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0810 04:07:11.783066 (+ 326us) server_negotiation.cc:299] Negotiation successful
0810 04:07:11.783131 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 04:07:11.783486 8680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:11.774990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:43401 (local address 127.31.32.2:49387)
0810 04:07:11.775142 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0810 04:07:11.775174 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0810 04:07:11.775598 (+ 424us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:07:11.775820 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:07:11.775823 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:07:11.776409 (+ 586us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:07:11.776581 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:11.776586 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:11.777207 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:11.777208 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:11.777974 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:11.777976 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:11.778295 (+ 319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:11.778296 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:11.778387 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:11.778770 (+ 383us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:07:11.778778 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:07:11.780108 (+ 1330us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:07:11.781884 (+ 1776us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:07:11.781891 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:07:11.781893 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:07:11.782145 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:07:11.782363 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:07:11.782364 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:07:11.782364 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:07:11.782510 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:07:11.782752 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:07:11.782758 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:07:11.782931 (+ 173us) client_negotiation.cc:769] Sending connection context
0810 04:07:11.783045 (+ 114us) client_negotiation.cc:240] Negotiation successful
0810 04:07:11.783215 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":96,"threads_started":1}
I20260810 04:07:11.783998 8678 heartbeater.cc:347] Connected to a master server at 127.31.32.62:43401
I20260810 04:07:11.784159 8678 heartbeater.cc:464] Registering TS with master...
I20260810 04:07:11.784394 8678 heartbeater.cc:511] Master 127.31.32.62:43401 requested a full tablet report, sending...
I20260810 04:07:11.784837 8314 ts_manager.cc:194] Registered new tserver with Master: ecc296d748d741069f4debb361e79664 (127.31.32.2:33853)
I20260810 04:07:11.785342 8314 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.32.2@KRBTEST.COM'} at 127.31.32.2:49387
WARNING: no policy specified for kudu/127.31.32.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.32.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.32.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.3.keytab.
Entry for principal kudu/127.31.32.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.3.keytab.
I20260810 04:07:11.807284 31872 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskws89Io/build/release/bin/kudu
/tmp/dist-test-taskws89Io/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.32.3:0
--local_ip_for_outbound_sockets=127.31.32.3
--webserver_interface=127.31.32.3
--webserver_port=0
--tserver_master_addrs=127.31.32.62:43401
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.3.keytab
--principal=kudu/127.31.32.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.32.20:41699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:07:11.889247 8683 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:07:11.889447 8683 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:07:11.889470 8683 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:07:11.890602 8683 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:07:11.890643 8683 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:07:11.890803 8683 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.32.3
I20260810 04:07:11.892409 8683 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.32.20:41699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.3.keytab
--principal=kudu/127.31.32.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.32.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.32.3
--webserver_port=0
--tserver_master_addrs=127.31.32.62:43401
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8683
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.32.3
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:07:11.892644 8683 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:07:11.892871 8683 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:07:11.897799 8691 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:07:11.897858 8689 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:07:11.897818 8688 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:07:11.898031 8683 server_base.cc:1034] running on GCE node
I20260810 04:07:11.898252 8683 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:07:11.898453 8683 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:07:11.899636 8683 hybrid_clock.cc:648] HybridClock initialized: now 1786334831899620 us; error 32 us; skew 500 ppm
Aug 10 04:07:11 dist-test-slave-8w9v krb5kdc[8265](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334831, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:07:11.902137 8683 init.cc:377] Logged in from keytab as kudu/127.31.32.3@KRBTEST.COM (short username kudu)
I20260810 04:07:11.902537 8683 webserver.cc:460] Webserver started at http://127.31.32.3:40109/ using document root <none> and password file <none>
I20260810 04:07:11.902796 8683 fs_manager.cc:359] Metadata directory not provided
I20260810 04:07:11.902871 8683 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:07:11.903019 8683 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:07:11.904064 8683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/minicluster-data/ts-2/data/instance:
uuid: "5349d58a2cdd4d4cae012b6e9076871f"
format_stamp: "Formatted at 2026-08-10 04:07:11 on dist-test-slave-8w9v"
server_key: "846ab1d6ca8d945ca7468f78416d56cd"
server_key_iv: "2763303aaf56fe272cf0c526758c8cae"
server_key_version: "encryptionkey@0"
I20260810 04:07:11.904508 8683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/minicluster-data/ts-2/wal/instance:
uuid: "5349d58a2cdd4d4cae012b6e9076871f"
format_stamp: "Formatted at 2026-08-10 04:07:11 on dist-test-slave-8w9v"
server_key: "846ab1d6ca8d945ca7468f78416d56cd"
server_key_iv: "2763303aaf56fe272cf0c526758c8cae"
server_key_version: "encryptionkey@0"
I20260810 04:07:11.905860 8683 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:07:11.906545 8698 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:07:11.906719 8683 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 04:07:11.906823 8683 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/minicluster-data/ts-2/data,/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/minicluster-data/ts-2/wal
uuid: "5349d58a2cdd4d4cae012b6e9076871f"
format_stamp: "Formatted at 2026-08-10 04:07:11 on dist-test-slave-8w9v"
server_key: "846ab1d6ca8d945ca7468f78416d56cd"
server_key_iv: "2763303aaf56fe272cf0c526758c8cae"
server_key_version: "encryptionkey@0"
I20260810 04:07:11.906903 8683 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:07:11.924706 8683 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:07:11.926434 8683 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:07:11.926592 8683 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:07:11.926808 8683 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:07:11.927112 8683 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:07:11.927165 8683 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:07:11.927199 8683 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:07:11.927245 8683 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:07:11.933341 8683 rpc_server.cc:307] RPC server started. Bound to: 127.31.32.3:40901
I20260810 04:07:11.933415 8822 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.32.3:40901 every 8 connection(s)
I20260810 04:07:11.933727 8683 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/minicluster-data/ts-2/data/info.pb
Aug 10 04:07:11 dist-test-slave-8w9v krb5kdc[8265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334831, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.3@KRBTEST.COM for kudu/127.31.32.62@KRBTEST.COM
I20260810 04:07:11.942127 31872 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskws89Io/build/release/bin/kudu as pid 8683
I20260810 04:07:11.942245 31872 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/minicluster-data/ts-2/wal/instance
I20260810 04:07:11.942461 31872 external_mini_cluster.cc:1428] Setting key ae409bfce0a7be768d6ca5526b477ce7
I20260810 04:07:11.942529 8378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:11.934448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.3:51931 (local address 127.31.32.62:43401)
0810 04:07:11.934549 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0810 04:07:11.934551 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:07:11.934712 (+ 161us) server_negotiation.cc:407] Connection header received
0810 04:07:11.935226 (+ 514us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:07:11.935226 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:07:11.935236 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:07:11.935284 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:07:11.936216 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:11.936695 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:11.937594 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:11.937763 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:11.937812 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:11.939777 (+ 1965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:07:11.939779 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:07:11.939779 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:07:11.939785 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:07:11.941183 (+ 1398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:07:11.941543 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:07:11.941544 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:07:11.941544 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:07:11.941595 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:07:11.941864 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:07:11.941865 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:07:11.941865 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:07:11.941998 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:07:11.942071 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0810 04:07:11.942349 (+ 278us) server_negotiation.cc:299] Negotiation successful
0810 04:07:11.942430 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 04:07:11.942689 8825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:11.934384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:43401 (local address 127.31.32.3:51931)
0810 04:07:11.934566 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0810 04:07:11.934611 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 04:07:11.935131 (+ 520us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:07:11.935350 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:07:11.935353 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:07:11.935903 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:07:11.936095 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:11.936100 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:11.936771 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:11.936772 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:11.937527 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:11.937529 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:11.937833 (+ 304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:11.937834 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:11.937926 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:11.938321 (+ 395us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:07:11.938328 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:07:11.939694 (+ 1366us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:07:11.941257 (+ 1563us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:07:11.941262 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:07:11.941264 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:07:11.941478 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:07:11.941653 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:07:11.941654 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:07:11.941654 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:07:11.941804 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:07:11.942067 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:07:11.942072 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:07:11.942191 (+ 119us) client_negotiation.cc:769] Sending connection context
0810 04:07:11.942306 (+ 115us) client_negotiation.cc:240] Negotiation successful
0810 04:07:11.942446 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":89,"threads_started":1}
I20260810 04:07:11.943113 8823 heartbeater.cc:347] Connected to a master server at 127.31.32.62:43401
I20260810 04:07:11.943202 8823 heartbeater.cc:464] Registering TS with master...
I20260810 04:07:11.943392 8823 heartbeater.cc:511] Master 127.31.32.62:43401 requested a full tablet report, sending...
I20260810 04:07:11.943783 8314 ts_manager.cc:194] Registered new tserver with Master: 5349d58a2cdd4d4cae012b6e9076871f (127.31.32.3:40901)
I20260810 04:07:11.944178 8314 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.32.3@KRBTEST.COM'} at 127.31.32.3:51931
I20260810 04:07:11.954219 31872 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.31.32.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.32.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.32.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/krb5kdc/oryx_127.31.32.1.keytab.
Entry for principal oryx/127.31.32.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/krb5kdc/oryx_127.31.32.1.keytab.
WARNING: no policy specified for oryx/127.31.32.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.32.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.32.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/krb5kdc/oryx_127.31.32.2.keytab.
Entry for principal oryx/127.31.32.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/krb5kdc/oryx_127.31.32.2.keytab.
WARNING: no policy specified for oryx/127.31.32.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.32.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.32.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/krb5kdc/oryx_127.31.32.3.keytab.
Entry for principal oryx/127.31.32.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/krb5kdc/oryx_127.31.32.3.keytab.
I20260810 04:07:12.024869 31872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskws89Io/build/release/bin/kudu with pid 8382
I20260810 04:07:12.033035 31872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskws89Io/build/release/bin/kudu with pid 8531
I20260810 04:07:12.040426 31872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskws89Io/build/release/bin/kudu with pid 8683
I20260810 04:07:12.046540 31872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskws89Io/build/release/bin/kudu with pid 8279
I20260810 04:07:12.055226 31872 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskws89Io/build/release/bin/kudu
/tmp/dist-test-taskws89Io/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.32.62:43401
--webserver_interface=127.31.32.62
--webserver_port=34481
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.62.keytab
--principal=kudu/127.31.32.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.32.20:41699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.32.62:43401
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:07:12.140789 8834 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:07:12.141052 8834 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:07:12.141086 8834 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:07:12.142267 8834 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:07:12.142328 8834 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:07:12.142352 8834 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:07:12.142382 8834 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:07:12.142408 8834 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:07:12.144567 8834 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.32.20:41699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.32.62:43401
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.62.keytab
--principal=kudu/127.31.32.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.32.62:43401
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.32.62
--webserver_port=34481
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8834
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:07:12.144819 8834 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:07:12.145071 8834 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:07:12.151382 8840 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:07:12.151472 8839 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:07:12.151546 8834 server_base.cc:1034] running on GCE node
W20260810 04:07:12.151650 8842 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:07:12.152108 8834 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:07:12.152494 8834 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:07:12.153680 8834 hybrid_clock.cc:648] HybridClock initialized: now 1786334832153660 us; error 36 us; skew 500 ppm
Aug 10 04:07:12 dist-test-slave-8w9v krb5kdc[8265](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334832, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:07:12.156949 8834 init.cc:377] Logged in from keytab as kudu/127.31.32.62@KRBTEST.COM (short username kudu)
I20260810 04:07:12.157490 8834 webserver.cc:460] Webserver started at http://127.31.32.62:34481/ using document root <none> and password file <none>
I20260810 04:07:12.157752 8834 fs_manager.cc:359] Metadata directory not provided
I20260810 04:07:12.157850 8834 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:07:12.159804 8834 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260810 04:07:12.160718 8850 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:07:12.160920 8834 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260810 04:07:12.161020 8834 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/minicluster-data/master-0/data,/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/minicluster-data/master-0/wal
uuid: "7e1ed946ff53405b95e9c249a76361e8"
format_stamp: "Formatted at 2026-08-10 04:07:11 on dist-test-slave-8w9v"
server_key: "d9464e398b80b59952a6589d8eaa715a"
server_key_iv: "297178dafee0b7f5873da9275babae7c"
server_key_version: "encryptionkey@0"
I20260810 04:07:12.161509 8834 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:07:12.189765 8834 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:07:12.192059 8834 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:07:12.192256 8834 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:07:12.196059 8834 rpc_server.cc:307] RPC server started. Bound to: 127.31.32.62:43401
I20260810 04:07:12.196159 8907 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.32.62:43401 every 8 connection(s)
I20260810 04:07:12.196512 8834 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/minicluster-data/master-0/data/info.pb
I20260810 04:07:12.197944 8908 sys_catalog.cc:263] Verifying existing consensus state
I20260810 04:07:12.198604 8908 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e1ed946ff53405b95e9c249a76361e8: Bootstrap starting.
I20260810 04:07:12.200295 31872 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskws89Io/build/release/bin/kudu as pid 8834
I20260810 04:07:12.200626 31872 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskws89Io/build/release/bin/kudu
/tmp/dist-test-taskws89Io/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.32.1:40891
--local_ip_for_outbound_sockets=127.31.32.1
--tserver_master_addrs=127.31.32.62:43401
--webserver_port=38341
--webserver_interface=127.31.32.1
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.1.keytab
--principal=kudu/127.31.32.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.32.20:41699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.31.32.1
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/krb5kdc/oryx_127.31.32.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 04:07:12.207469 8908 log.cc:824] T 00000000000000000000000000000000 P 7e1ed946ff53405b95e9c249a76361e8: Log is configured to *not* fsync() on all Append() calls
I20260810 04:07:12.209167 8908 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e1ed946ff53405b95e9c249a76361e8: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 04:07:12.209462 8908 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e1ed946ff53405b95e9c249a76361e8: Bootstrap complete.
I20260810 04:07:12.211740 8908 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7e1ed946ff53405b95e9c249a76361e8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e1ed946ff53405b95e9c249a76361e8" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 43401 } }
I20260810 04:07:12.211993 8908 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7e1ed946ff53405b95e9c249a76361e8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e1ed946ff53405b95e9c249a76361e8, State: Initialized, Role: FOLLOWER
I20260810 04:07:12.212227 8908 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7e1ed946ff53405b95e9c249a76361e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e1ed946ff53405b95e9c249a76361e8" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 43401 } }
I20260810 04:07:12.212338 8908 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7e1ed946ff53405b95e9c249a76361e8 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 04:07:12.212395 8908 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7e1ed946ff53405b95e9c249a76361e8 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 04:07:12.212452 8908 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7e1ed946ff53405b95e9c249a76361e8 [term 1 FOLLOWER]: Advancing to term 2
I20260810 04:07:12.213429 8908 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7e1ed946ff53405b95e9c249a76361e8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e1ed946ff53405b95e9c249a76361e8" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 43401 } }
I20260810 04:07:12.213548 8908 leader_election.cc:302] T 00000000000000000000000000000000 P 7e1ed946ff53405b95e9c249a76361e8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7e1ed946ff53405b95e9c249a76361e8; no voters:
I20260810 04:07:12.213732 8908 leader_election.cc:288] T 00000000000000000000000000000000 P 7e1ed946ff53405b95e9c249a76361e8 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260810 04:07:12.213795 8912 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7e1ed946ff53405b95e9c249a76361e8 [term 2 FOLLOWER]: Leader election won for term 2
I20260810 04:07:12.213958 8912 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7e1ed946ff53405b95e9c249a76361e8 [term 2 LEADER]: Becoming Leader. State: Replica: 7e1ed946ff53405b95e9c249a76361e8, State: Running, Role: LEADER
I20260810 04:07:12.214010 8908 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7e1ed946ff53405b95e9c249a76361e8 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:07:12.214051 8912 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7e1ed946ff53405b95e9c249a76361e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e1ed946ff53405b95e9c249a76361e8" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 43401 } }
I20260810 04:07:12.214314 8912 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e1ed946ff53405b95e9c249a76361e8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7e1ed946ff53405b95e9c249a76361e8. Latest consensus state: current_term: 2 leader_uuid: "7e1ed946ff53405b95e9c249a76361e8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e1ed946ff53405b95e9c249a76361e8" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 43401 } } }
I20260810 04:07:12.214315 8913 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e1ed946ff53405b95e9c249a76361e8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "7e1ed946ff53405b95e9c249a76361e8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e1ed946ff53405b95e9c249a76361e8" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 43401 } } }
I20260810 04:07:12.214439 8912 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e1ed946ff53405b95e9c249a76361e8 [sys.catalog]: This master's current role is: LEADER
I20260810 04:07:12.214483 8913 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e1ed946ff53405b95e9c249a76361e8 [sys.catalog]: This master's current role is: LEADER
I20260810 04:07:12.215014 8916 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 04:07:12.215528 8916 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 04:07:12.215680 8916 catalog_manager.cc:1259] Loaded cluster ID: 1e88a24b93054b5a9d1da3c33f382054
I20260810 04:07:12.215890 8916 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 04:07:12.216759 8916 catalog_manager.cc:1504] Loading token signing keys...
I20260810 04:07:12.216950 8916 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 7e1ed946ff53405b95e9c249a76361e8: Loaded TSK: 0
I20260810 04:07:12.217164 8916 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 04:07:12.298324 8910 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:07:12.298521 8910 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:07:12.298543 8910 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:07:12.299659 8910 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:07:12.299698 8910 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:07:12.299723 8910 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.32.1
I20260810 04:07:12.301173 8910 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.32.20:41699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/krb5kdc/oryx_127.31.32.1.keytab
--principal=oryx/127.31.32.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.32.1:40891
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.32.1
--webserver_port=38341
--tserver_master_addrs=127.31.32.62:43401
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8910
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.32.1
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:07:12.301398 8910 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:07:12.301644 8910 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:07:12.306563 8941 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:07:12.306543 8937 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:07:12.306751 8910 server_base.cc:1034] running on GCE node
W20260810 04:07:12.306854 8939 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:07:12.307106 8910 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:07:12.307394 8910 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:07:12.308575 8910 hybrid_clock.cc:648] HybridClock initialized: now 1786334832308553 us; error 36 us; skew 500 ppm
Aug 10 04:07:12 dist-test-slave-8w9v krb5kdc[8265](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334832, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.32.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:07:12.311494 8910 init.cc:377] Logged in from keytab as oryx/127.31.32.1@KRBTEST.COM (short username oryx)
I20260810 04:07:12.311947 8910 webserver.cc:460] Webserver started at http://127.31.32.1:38341/ using document root <none> and password file <none>
I20260810 04:07:12.312229 8910 fs_manager.cc:359] Metadata directory not provided
I20260810 04:07:12.312312 8910 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:07:12.313678 8910 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260810 04:07:12.314350 8948 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:07:12.314527 8910 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 04:07:12.314621 8910 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/minicluster-data/ts-0/data,/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/minicluster-data/ts-0/wal
uuid: "782397279dd24fbe8ef43831f3dcb8e5"
format_stamp: "Formatted at 2026-08-10 04:07:11 on dist-test-slave-8w9v"
server_key: "1b6df4d1ecce7874ce0f04f2ce03729c"
server_key_iv: "dd7224328f3497ad788f1a66a4ac340d"
server_key_version: "encryptionkey@0"
I20260810 04:07:12.314949 8910 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:07:12.332319 8910 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:07:12.334579 8910 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:07:12.334738 8910 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:07:12.334966 8910 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:07:12.335314 8910 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:07:12.335368 8910 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:07:12.335407 8910 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:07:12.335434 8910 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:07:12.341738 8910 rpc_server.cc:307] RPC server started. Bound to: 127.31.32.1:40891
I20260810 04:07:12.341787 9075 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.32.1:40891 every 8 connection(s)
I20260810 04:07:12.342103 8910 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/minicluster-data/ts-0/data/info.pb
I20260810 04:07:12.346884 31872 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskws89Io/build/release/bin/kudu as pid 8910
I20260810 04:07:12.347190 31872 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskws89Io/build/release/bin/kudu
/tmp/dist-test-taskws89Io/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.32.2:33853
--local_ip_for_outbound_sockets=127.31.32.2
--tserver_master_addrs=127.31.32.62:43401
--webserver_port=41135
--webserver_interface=127.31.32.2
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.2.keytab
--principal=kudu/127.31.32.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.32.20:41699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.31.32.2
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/krb5kdc/oryx_127.31.32.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 04:07:12 dist-test-slave-8w9v krb5kdc[8265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.32.1@KRBTEST.COM for oryx/127.31.32.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 04:07:12 dist-test-slave-8w9v krb5kdc[8265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.32.1@KRBTEST.COM for oryx/127.31.32.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 04:07:12.351956 9078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:12.342786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:43401 (local address 127.31.32.1:47841)
0810 04:07:12.342951 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0810 04:07:12.342989 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0810 04:07:12.343442 (+ 453us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:07:12.343932 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:07:12.343935 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:07:12.344515 (+ 580us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:07:12.344690 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:12.344694 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:12.346211 (+ 1517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:12.346212 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:12.347315 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:12.347319 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:12.348236 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:12.348237 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:12.348373 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:12.348920 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:07:12.348929 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:07:12.351704 (+ 2775us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.32.62:43401: Server oryx/127.31.32.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":96,"threads_started":1}
W20260810 04:07:12.352221 9076 heartbeater.cc:644] Failed to heartbeat to 127.31.32.62:43401 (0 consecutive failures): Not authorized: Failed to ping master at 127.31.32.62:43401: Client connection negotiation failed: client connection to 127.31.32.62:43401: Server oryx/127.31.32.62@KRBTEST.COM not found in Kerberos database
W20260810 04:07:12.352411 9079 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:07:12.342865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.1:47841 (local address 127.31.32.62:43401)
0810 04:07:12.343043 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0810 04:07:12.343051 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 04:07:12.343088 (+ 37us) server_negotiation.cc:407] Connection header received
0810 04:07:12.343669 (+ 581us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:07:12.343676 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:07:12.343730 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:07:12.343893 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:07:12.344817 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:12.346118 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:12.347410 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:12.348034 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:12.348155 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:12.352235 (+ 4080us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.32.1:47841: BlockingRecv error: recv got EOF from 127.31.32.1:47841 (error 108)
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":82,"threads_started":1}
Aug 10 04:07:12 dist-test-slave-8w9v krb5kdc[8265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.32.1@KRBTEST.COM for oryx/127.31.32.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 04:07:12 dist-test-slave-8w9v krb5kdc[8265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.32.1@KRBTEST.COM for oryx/127.31.32.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 04:07:12.361681 9078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:12.352448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:43401 (local address 127.31.32.1:59515)
0810 04:07:12.356201 (+ 3753us) negotiation.cc:107] Waiting for socket to connect
0810 04:07:12.356213 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 04:07:12.356268 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:07:12.356462 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:07:12.356463 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:07:12.356621 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:07:12.356678 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:12.356681 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:12.357677 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:12.357678 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:12.358576 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:12.358579 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:12.359176 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:12.359177 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:12.359246 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:12.359735 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:07:12.359740 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:07:12.361636 (+ 1896us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.32.62:43401: Server oryx/127.31.32.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":3732}
W20260810 04:07:12.361796 9079 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:07:12.352515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.1:59515 (local address 127.31.32.62:43401)
0810 04:07:12.352585 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0810 04:07:12.352588 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:07:12.356326 (+ 3738us) server_negotiation.cc:407] Connection header received
0810 04:07:12.356351 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:07:12.356352 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:07:12.356366 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:07:12.356417 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:07:12.356768 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:12.357574 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:12.358691 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:12.359053 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:12.359119 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:12.361735 (+ 2616us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.32.1:59515: BlockingRecv error: recv got EOF from 127.31.32.1:59515 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Aug 10 04:07:12 dist-test-slave-8w9v krb5kdc[8265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.32.1@KRBTEST.COM for oryx/127.31.32.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 04:07:12 dist-test-slave-8w9v krb5kdc[8265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.32.1@KRBTEST.COM for oryx/127.31.32.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 04:07:12.366150 9078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:12.362040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:43401 (local address 127.31.32.1:51303)
0810 04:07:12.362222 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0810 04:07:12.362235 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 04:07:12.362292 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:07:12.362429 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:07:12.362430 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:07:12.362547 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:07:12.362571 (+ 24us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:12.362572 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:12.363141 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:12.363142 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:12.363601 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:12.363603 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:12.363945 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:12.363946 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:12.363996 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:12.364416 (+ 420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:07:12.364420 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:07:12.366070 (+ 1650us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.32.62:43401: Server oryx/127.31.32.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":122,"threads_started":1}
W20260810 04:07:12.366205 9079 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:07:12.362105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.1:51303 (local address 127.31.32.62:43401)
0810 04:07:12.362184 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0810 04:07:12.362186 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:07:12.362305 (+ 119us) server_negotiation.cc:407] Connection header received
0810 04:07:12.362346 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:07:12.362347 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:07:12.362357 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:07:12.362426 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:07:12.362653 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:12.363067 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:12.363694 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:12.363876 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:12.363947 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:12.366145 (+ 2198us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.32.1:51303: BlockingRecv error: recv got EOF from 127.31.32.1:51303 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260810 04:07:12.366267 9076 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 04:07:12.430969 9080 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:07:12.431190 9080 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:07:12.431227 9080 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:07:12.432400 9080 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:07:12.432464 9080 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:07:12.432507 9080 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.32.2
I20260810 04:07:12.433899 9080 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.32.20:41699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/krb5kdc/oryx_127.31.32.2.keytab
--principal=oryx/127.31.32.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.32.2:33853
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.32.2
--webserver_port=41135
--tserver_master_addrs=127.31.32.62:43401
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9080
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.32.2
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:07:12.434165 9080 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:07:12.434402 9080 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:07:12.439318 9087 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:07:12.439484 9080 server_base.cc:1034] running on GCE node
W20260810 04:07:12.439306 9090 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:07:12.439321 9088 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:07:12.439816 9080 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:07:12.440073 9080 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:07:12.441283 9080 hybrid_clock.cc:648] HybridClock initialized: now 1786334832441263 us; error 38 us; skew 500 ppm
Aug 10 04:07:12 dist-test-slave-8w9v krb5kdc[8265](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334832, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.32.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:07:12.443941 9080 init.cc:377] Logged in from keytab as oryx/127.31.32.2@KRBTEST.COM (short username oryx)
I20260810 04:07:12.444362 9080 webserver.cc:460] Webserver started at http://127.31.32.2:41135/ using document root <none> and password file <none>
I20260810 04:07:12.444553 9080 fs_manager.cc:359] Metadata directory not provided
I20260810 04:07:12.444602 9080 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:07:12.445791 9080 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:07:12.446316 9098 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:07:12.446481 9080 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 04:07:12.446578 9080 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/minicluster-data/ts-1/data,/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/minicluster-data/ts-1/wal
uuid: "ecc296d748d741069f4debb361e79664"
format_stamp: "Formatted at 2026-08-10 04:07:11 on dist-test-slave-8w9v"
server_key: "4e0e42e15a87e9c118b7d1577120970d"
server_key_iv: "68cc980deb195df05187440780488fa6"
server_key_version: "encryptionkey@0"
I20260810 04:07:12.446900 9080 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:07:12.462059 9080 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:07:12.463907 9080 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:07:12.464056 9080 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:07:12.464313 9080 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:07:12.464598 9080 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:07:12.464651 9080 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:07:12.464708 9080 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:07:12.464733 9080 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:07:12.470387 9080 rpc_server.cc:307] RPC server started. Bound to: 127.31.32.2:33853
I20260810 04:07:12.470463 9221 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.32.2:33853 every 8 connection(s)
I20260810 04:07:12.470786 9080 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/minicluster-data/ts-1/data/info.pb
I20260810 04:07:12.473580 31872 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskws89Io/build/release/bin/kudu as pid 9080
I20260810 04:07:12.473884 31872 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskws89Io/build/release/bin/kudu
/tmp/dist-test-taskws89Io/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.32.3:40901
--local_ip_for_outbound_sockets=127.31.32.3
--tserver_master_addrs=127.31.32.62:43401
--webserver_port=40109
--webserver_interface=127.31.32.3
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.3.keytab
--principal=kudu/127.31.32.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.32.20:41699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.31.32.3
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/krb5kdc/oryx_127.31.32.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 04:07:12 dist-test-slave-8w9v krb5kdc[8265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.32.2@KRBTEST.COM for oryx/127.31.32.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 04:07:12 dist-test-slave-8w9v krb5kdc[8265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.32.2@KRBTEST.COM for oryx/127.31.32.62@KRBTEST.COM, Server not found in Kerberos database
W20260810 04:07:12.478665 9079 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:07:12.471462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.2:34869 (local address 127.31.32.62:43401)
0810 04:07:12.471506 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0810 04:07:12.471508 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:07:12.471698 (+ 190us) server_negotiation.cc:407] Connection header received
0810 04:07:12.472245 (+ 547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:07:12.472247 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:07:12.472263 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:07:12.472308 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:07:12.473244 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:12.473832 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:12.474913 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:12.475137 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:12.475221 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:12.478618 (+ 3397us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.32.2:34869: BlockingRecv error: recv got EOF from 127.31.32.2:34869 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
I20260810 04:07:12.478770 9224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:12.471419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:43401 (local address 127.31.32.2:34869)
0810 04:07:12.471554 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0810 04:07:12.471617 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0810 04:07:12.472103 (+ 486us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:07:12.472388 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:07:12.472392 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:07:12.472936 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:07:12.473145 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:12.473149 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:12.473939 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:12.473940 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:12.474821 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:12.474823 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:12.475234 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:12.475235 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:12.475357 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:12.475837 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:07:12.475845 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:07:12.478541 (+ 2696us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.32.62:43401: Server oryx/127.31.32.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":66,"threads_started":1}
W20260810 04:07:12.479086 9222 heartbeater.cc:644] Failed to heartbeat to 127.31.32.62:43401 (0 consecutive failures): Not authorized: Failed to ping master at 127.31.32.62:43401: Client connection negotiation failed: client connection to 127.31.32.62:43401: Server oryx/127.31.32.62@KRBTEST.COM not found in Kerberos database
Aug 10 04:07:12 dist-test-slave-8w9v krb5kdc[8265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.32.2@KRBTEST.COM for oryx/127.31.32.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 04:07:12 dist-test-slave-8w9v krb5kdc[8265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.32.2@KRBTEST.COM for oryx/127.31.32.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 04:07:12.483733 9224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:12.479364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:43401 (local address 127.31.32.2:37969)
0810 04:07:12.479415 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0810 04:07:12.479427 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 04:07:12.479523 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:07:12.479658 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:07:12.479658 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:07:12.479786 (+ 128us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:07:12.479826 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:12.479827 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:12.480507 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:12.480508 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:12.481026 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:12.481028 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:12.481366 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:12.481367 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:12.481431 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:12.481832 (+ 401us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:07:12.481837 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:07:12.483657 (+ 1820us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.32.62:43401: Server oryx/127.31.32.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":34}
W20260810 04:07:12.483786 9079 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:07:12.479450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.2:37969 (local address 127.31.32.62:43401)
0810 04:07:12.479489 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0810 04:07:12.479491 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:07:12.479499 (+ 8us) server_negotiation.cc:407] Connection header received
0810 04:07:12.479587 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:07:12.479588 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:07:12.479598 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:07:12.479653 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:07:12.479900 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:12.480402 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:12.481117 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:12.481293 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:12.481366 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:12.483724 (+ 2358us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.32.2:37969: BlockingRecv error: recv got EOF from 127.31.32.2:37969 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
Aug 10 04:07:12 dist-test-slave-8w9v krb5kdc[8265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.32.2@KRBTEST.COM for oryx/127.31.32.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 04:07:12 dist-test-slave-8w9v krb5kdc[8265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.32.2@KRBTEST.COM for oryx/127.31.32.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 04:07:12.488029 9224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:12.483976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:43401 (local address 127.31.32.2:56431)
0810 04:07:12.484111 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0810 04:07:12.484160 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0810 04:07:12.484206 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:07:12.484353 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:07:12.484354 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:07:12.484470 (+ 116us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:07:12.484494 (+ 24us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:12.484495 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:12.485063 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:12.485064 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:12.485527 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:12.485529 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:12.485884 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:12.485884 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:12.485950 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:12.486342 (+ 392us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:07:12.486346 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:07:12.487963 (+ 1617us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.32.62:43401: Server oryx/127.31.32.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":87,"threads_started":1}
W20260810 04:07:12.488102 9079 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:07:12.484018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.2:56431 (local address 127.31.32.62:43401)
0810 04:07:12.484052 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0810 04:07:12.484054 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:07:12.484193 (+ 139us) server_negotiation.cc:407] Connection header received
0810 04:07:12.484260 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:07:12.484261 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:07:12.484269 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:07:12.484338 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:07:12.484566 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:12.484986 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:12.485604 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:12.485779 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:12.485882 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:12.488036 (+ 2154us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.32.2:56431: BlockingRecv error: recv got EOF from 127.31.32.2:56431 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260810 04:07:12.488162 9222 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 04:07:12.556941 9225 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:07:12.557123 9225 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:07:12.557144 9225 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:07:12.558270 9225 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:07:12.558306 9225 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:07:12.558357 9225 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.32.3
I20260810 04:07:12.559760 9225 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.32.20:41699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/krb5kdc/oryx_127.31.32.3.keytab
--principal=oryx/127.31.32.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.32.3:40901
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.32.3
--webserver_port=40109
--tserver_master_addrs=127.31.32.62:43401
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9225
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.32.3
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:07:12.559996 9225 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:07:12.560269 9225 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:07:12.564939 9231 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:07:12.564978 9232 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:07:12.564966 9234 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:07:12.565229 9225 server_base.cc:1034] running on GCE node
I20260810 04:07:12.565475 9225 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:07:12.565703 9225 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:07:12.566862 9225 hybrid_clock.cc:648] HybridClock initialized: now 1786334832566845 us; error 31 us; skew 500 ppm
Aug 10 04:07:12 dist-test-slave-8w9v krb5kdc[8265](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334832, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.32.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:07:12.569509 9225 init.cc:377] Logged in from keytab as oryx/127.31.32.3@KRBTEST.COM (short username oryx)
I20260810 04:07:12.569955 9225 webserver.cc:460] Webserver started at http://127.31.32.3:40109/ using document root <none> and password file <none>
I20260810 04:07:12.570143 9225 fs_manager.cc:359] Metadata directory not provided
I20260810 04:07:12.570194 9225 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:07:12.571386 9225 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260810 04:07:12.571892 9241 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:07:12.572077 9225 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 04:07:12.572191 9225 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/minicluster-data/ts-2/data,/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/minicluster-data/ts-2/wal
uuid: "5349d58a2cdd4d4cae012b6e9076871f"
format_stamp: "Formatted at 2026-08-10 04:07:11 on dist-test-slave-8w9v"
server_key: "846ab1d6ca8d945ca7468f78416d56cd"
server_key_iv: "2763303aaf56fe272cf0c526758c8cae"
server_key_version: "encryptionkey@0"
I20260810 04:07:12.572484 9225 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:07:12.582063 9225 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:07:12.583683 9225 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:07:12.583834 9225 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:07:12.584039 9225 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:07:12.584342 9225 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:07:12.584376 9225 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:07:12.584395 9225 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:07:12.584439 9225 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:07:12.590157 9225 rpc_server.cc:307] RPC server started. Bound to: 127.31.32.3:40901
I20260810 04:07:12.590209 9371 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.32.3:40901 every 8 connection(s)
I20260810 04:07:12.590514 9225 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786334794676589-31872-0/minicluster-data/ts-2/data/info.pb
Aug 10 04:07:12 dist-test-slave-8w9v krb5kdc[8265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.32.3@KRBTEST.COM for oryx/127.31.32.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 04:07:12 dist-test-slave-8w9v krb5kdc[8265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.32.3@KRBTEST.COM for oryx/127.31.32.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 04:07:12.597365 9374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:12.591172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:43401 (local address 127.31.32.3:51147)
0810 04:07:12.591314 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0810 04:07:12.591348 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0810 04:07:12.591804 (+ 456us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:07:12.592061 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:07:12.592065 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:07:12.592610 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:07:12.592778 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:12.592783 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:12.593462 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:12.593463 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:12.594240 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:12.594243 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:12.594634 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:12.594635 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:12.594750 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:12.595233 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:07:12.595241 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:07:12.597247 (+ 2006us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.32.62:43401: Server oryx/127.31.32.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":83,"threads_started":1}
W20260810 04:07:12.597378 9079 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:07:12.591194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.3:51147 (local address 127.31.32.62:43401)
0810 04:07:12.591239 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0810 04:07:12.591241 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:07:12.591435 (+ 194us) server_negotiation.cc:407] Connection header received
0810 04:07:12.591918 (+ 483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:07:12.591920 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:07:12.591932 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:07:12.591993 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:07:12.592888 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:12.593368 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:12.594342 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:12.594539 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:12.594622 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:12.597324 (+ 2702us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.32.3:51147: BlockingRecv error: recv got EOF from 127.31.32.3:51147 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260810 04:07:12.597620 9372 heartbeater.cc:644] Failed to heartbeat to 127.31.32.62:43401 (0 consecutive failures): Not authorized: Failed to ping master at 127.31.32.62:43401: Client connection negotiation failed: client connection to 127.31.32.62:43401: Server oryx/127.31.32.62@KRBTEST.COM not found in Kerberos database
I20260810 04:07:12.600003 31872 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskws89Io/build/release/bin/kudu as pid 9225
Aug 10 04:07:12 dist-test-slave-8w9v krb5kdc[8265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.32.3@KRBTEST.COM for oryx/127.31.32.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 04:07:12 dist-test-slave-8w9v krb5kdc[8265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.32.3@KRBTEST.COM for oryx/127.31.32.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 04:07:12.601879 9374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:12.597829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:43401 (local address 127.31.32.3:52209)
0810 04:07:12.597880 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0810 04:07:12.597890 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 04:07:12.597928 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:07:12.598137 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:07:12.598138 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:07:12.598240 (+ 102us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:07:12.598280 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:12.598282 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:12.598811 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:12.598812 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:12.599294 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:12.599296 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:12.599610 (+ 314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:12.599611 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:12.599655 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:12.600009 (+ 354us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:07:12.600015 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:07:12.601840 (+ 1825us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.32.62:43401: Server oryx/127.31.32.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":36}
W20260810 04:07:12.601986 9079 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:07:12.597988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.3:52209 (local address 127.31.32.62:43401)
0810 04:07:12.598041 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0810 04:07:12.598043 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:07:12.598050 (+ 7us) server_negotiation.cc:407] Connection header received
0810 04:07:12.598066 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:07:12.598067 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:07:12.598074 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:07:12.598120 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:07:12.598350 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:12.598751 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:12.599395 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:12.599551 (+ 156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:12.599597 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:12.601929 (+ 2332us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.32.3:52209: BlockingRecv error: recv got EOF from 127.31.32.3:52209 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Aug 10 04:07:12 dist-test-slave-8w9v krb5kdc[8265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.32.3@KRBTEST.COM for oryx/127.31.32.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 04:07:12 dist-test-slave-8w9v krb5kdc[8265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.32.3@KRBTEST.COM for oryx/127.31.32.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 04:07:12.606415 9374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:12.602104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:43401 (local address 127.31.32.3:48101)
0810 04:07:12.602218 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0810 04:07:12.602227 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 04:07:12.602269 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:07:12.602391 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:07:12.602392 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:07:12.602489 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:07:12.602512 (+ 23us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:12.602513 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:12.603266 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:12.603267 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:12.603768 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:12.603770 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:12.604221 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:12.604222 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:12.604267 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:12.604636 (+ 369us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:07:12.604640 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:07:12.606341 (+ 1701us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.32.62:43401: Server oryx/127.31.32.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":78,"threads_started":1}
W20260810 04:07:12.606451 9079 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:07:12.602136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.3:48101 (local address 127.31.32.62:43401)
0810 04:07:12.602165 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0810 04:07:12.602167 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:07:12.602264 (+ 97us) server_negotiation.cc:407] Connection header received
0810 04:07:12.602318 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:07:12.602318 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:07:12.602327 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:07:12.602376 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:07:12.602577 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:12.603196 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:12.603828 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:12.604135 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:12.604202 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:12.606397 (+ 2195us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.32.3:48101: BlockingRecv error: recv got EOF from 127.31.32.3:48101 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260810 04:07:12.606630 9372 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260810 04:07:12.608079 9376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:12.600447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57676 (local address 127.31.32.62:43401)
0810 04:07:12.600572 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0810 04:07:12.600575 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:07:12.600665 (+ 90us) server_negotiation.cc:407] Connection header received
0810 04:07:12.600692 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:07:12.600693 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:07:12.600703 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:07:12.600774 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:07:12.601392 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:12.601956 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:12.603050 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:12.603278 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:12.603344 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:12.604718 (+ 1374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:07:12.604723 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:07:12.604723 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:07:12.604734 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:07:12.606397 (+ 1663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:07:12.607014 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:07:12.607017 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:07:12.607018 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:07:12.607114 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:07:12.607378 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:07:12.607379 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:07:12.607379 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:07:12.607611 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:07:12.607761 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0810 04:07:12.607874 (+ 113us) server_negotiation.cc:299] Negotiation successful
0810 04:07:12.607986 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":61,"threads_started":1}
Aug 10 04:07:13 dist-test-slave-8w9v krb5kdc[8265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.32.1@KRBTEST.COM for oryx/127.31.32.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 04:07:13 dist-test-slave-8w9v krb5kdc[8265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.32.1@KRBTEST.COM for oryx/127.31.32.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 04:07:13.371236 9378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:13.366649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:43401 (local address 127.31.32.1:34069)
0810 04:07:13.366797 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0810 04:07:13.366808 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 04:07:13.366863 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:07:13.367003 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:07:13.367004 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:07:13.367179 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:07:13.367238 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:13.367240 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:13.367864 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:13.367865 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:13.368432 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:13.368434 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:13.368755 (+ 321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:13.368756 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:13.368796 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:13.369227 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:07:13.369232 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:07:13.371165 (+ 1933us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.32.62:43401: Server oryx/127.31.32.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":87,"threads_started":1}
W20260810 04:07:13.371345 9379 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:07:13.366739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.1:34069 (local address 127.31.32.62:43401)
0810 04:07:13.366877 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0810 04:07:13.366879 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:07:13.366901 (+ 22us) server_negotiation.cc:407] Connection header received
0810 04:07:13.366926 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:07:13.366926 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:07:13.366936 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:07:13.366987 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:07:13.367301 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:13.367797 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:13.368500 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:13.368690 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:13.368743 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:13.371253 (+ 2510us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.32.1:34069: BlockingRecv error: recv got EOF from 127.31.32.1:34069 (error 108)
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":94,"threads_started":1}
Aug 10 04:07:13 dist-test-slave-8w9v krb5kdc[8265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.32.2@KRBTEST.COM for oryx/127.31.32.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 04:07:13 dist-test-slave-8w9v krb5kdc[8265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.32.2@KRBTEST.COM for oryx/127.31.32.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 04:07:13.493050 9380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:13.488514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:43401 (local address 127.31.32.2:40659)
0810 04:07:13.488655 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0810 04:07:13.488665 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 04:07:13.488722 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:07:13.488852 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:07:13.488853 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:07:13.489016 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:07:13.489067 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:13.489068 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:13.489679 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:13.489680 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:13.490236 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:13.490238 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:13.490560 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:13.490561 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:13.490600 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:13.491109 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:07:13.491114 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:07:13.492982 (+ 1868us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.32.62:43401: Server oryx/127.31.32.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":82,"threads_started":1}
W20260810 04:07:13.493183 9379 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:07:13.488580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.2:40659 (local address 127.31.32.62:43401)
0810 04:07:13.488626 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0810 04:07:13.488628 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:07:13.488723 (+ 95us) server_negotiation.cc:407] Connection header received
0810 04:07:13.488779 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:07:13.488779 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:07:13.488790 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:07:13.488832 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:07:13.489127 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:13.489612 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:13.490308 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:13.490498 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:13.490550 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:13.493078 (+ 2528us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.32.2:40659: BlockingRecv error: recv got EOF from 127.31.32.2:40659 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
Aug 10 04:07:13 dist-test-slave-8w9v krb5kdc[8265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.32.3@KRBTEST.COM for oryx/127.31.32.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 04:07:13 dist-test-slave-8w9v krb5kdc[8265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.32.3@KRBTEST.COM for oryx/127.31.32.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 04:07:13.611429 9382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:13.606977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:43401 (local address 127.31.32.3:44429)
0810 04:07:13.607116 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0810 04:07:13.607142 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0810 04:07:13.607194 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:07:13.607315 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:07:13.607316 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:07:13.607511 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:07:13.607561 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:13.607563 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:13.608183 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:13.608184 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:13.608703 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:13.608704 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:13.609036 (+ 332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:13.609037 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:13.609082 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:13.609514 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:07:13.609519 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:07:13.611363 (+ 1844us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.32.62:43401: Server oryx/127.31.32.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":84,"threads_started":1}
W20260810 04:07:13.611526 9379 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:07:13.607059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.3:44429 (local address 127.31.32.62:43401)
0810 04:07:13.607119 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0810 04:07:13.607121 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:07:13.607195 (+ 74us) server_negotiation.cc:407] Connection header received
0810 04:07:13.607235 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:07:13.607235 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:07:13.607246 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:07:13.607293 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:07:13.607620 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:13.608106 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:13.608778 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:13.608964 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:13.609017 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:13.611459 (+ 2442us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.32.3:44429: BlockingRecv error: recv got EOF from 127.31.32.3:44429 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Aug 10 04:07:14 dist-test-slave-8w9v krb5kdc[8265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.32.1@KRBTEST.COM for oryx/127.31.32.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 04:07:14 dist-test-slave-8w9v krb5kdc[8265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.32.1@KRBTEST.COM for oryx/127.31.32.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 04:07:14.376405 9383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:14.371868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:43401 (local address 127.31.32.1:33211)
0810 04:07:14.371997 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0810 04:07:14.372005 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0810 04:07:14.372047 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:07:14.372217 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:07:14.372217 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:07:14.372391 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:07:14.372443 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:14.372444 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:14.373047 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:14.373048 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:14.373542 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:14.373544 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:14.373905 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:14.373906 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:14.373982 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:14.374462 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:07:14.374467 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:07:14.376343 (+ 1876us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.32.62:43401: Server oryx/127.31.32.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":70,"threads_started":1}
W20260810 04:07:14.376533 9384 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:07:14.371921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.1:33211 (local address 127.31.32.62:43401)
0810 04:07:14.372039 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0810 04:07:14.372041 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:07:14.372068 (+ 27us) server_negotiation.cc:407] Connection header received
0810 04:07:14.372104 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:07:14.372105 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:07:14.372115 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:07:14.372218 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:07:14.372504 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:14.372980 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:14.373604 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:14.373784 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:14.373893 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:14.376446 (+ 2553us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.32.1:33211: BlockingRecv error: recv got EOF from 127.31.32.1:33211 (error 108)
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":67,"threads_started":1}
Aug 10 04:07:14 dist-test-slave-8w9v krb5kdc[8265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.32.2@KRBTEST.COM for oryx/127.31.32.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 04:07:14 dist-test-slave-8w9v krb5kdc[8265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.32.2@KRBTEST.COM for oryx/127.31.32.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 04:07:14.498243 9385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:14.493532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:43401 (local address 127.31.32.2:38803)
0810 04:07:14.493684 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0810 04:07:14.493695 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 04:07:14.493756 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:07:14.493884 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:07:14.493885 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:07:14.494054 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:07:14.494105 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:14.494106 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:14.494725 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:14.494725 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:14.495258 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:14.495260 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:14.495583 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:14.495584 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:14.495625 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:14.496179 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:07:14.496183 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:07:14.498173 (+ 1990us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.32.62:43401: Server oryx/127.31.32.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":78,"threads_started":1}
W20260810 04:07:14.498449 9384 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:07:14.493582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.2:38803 (local address 127.31.32.62:43401)
0810 04:07:14.493634 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0810 04:07:14.493637 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:07:14.493748 (+ 111us) server_negotiation.cc:407] Connection header received
0810 04:07:14.493804 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:07:14.493805 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:07:14.493815 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:07:14.493862 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:07:14.494171 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:14.494655 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:14.495335 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:14.495517 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:14.495567 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:14.498373 (+ 2806us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.32.2:38803: BlockingRecv error: recv got EOF from 127.31.32.2:38803 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
Aug 10 04:07:14 dist-test-slave-8w9v krb5kdc[8265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.32.3@KRBTEST.COM for oryx/127.31.32.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 04:07:14 dist-test-slave-8w9v krb5kdc[8265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.32.3@KRBTEST.COM for oryx/127.31.32.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 04:07:14.616896 9386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:14.612195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:43401 (local address 127.31.32.3:44089)
0810 04:07:14.612349 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0810 04:07:14.612362 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 04:07:14.612443 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:07:14.612649 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:07:14.612650 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:07:14.612867 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:07:14.612919 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:14.612922 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:14.613532 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:14.613533 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:14.614086 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:14.614087 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:14.614478 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:14.614479 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:14.614517 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:14.615090 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:07:14.615094 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:07:14.616833 (+ 1739us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.32.62:43401: Server oryx/127.31.32.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":91,"threads_started":1}
W20260810 04:07:14.617046 9384 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:07:14.612295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.3:44089 (local address 127.31.32.62:43401)
0810 04:07:14.612353 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0810 04:07:14.612356 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:07:14.612416 (+ 60us) server_negotiation.cc:407] Connection header received
0810 04:07:14.612519 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:07:14.612519 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:07:14.612535 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:07:14.612628 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:07:14.612990 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:14.613468 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:14.614174 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:14.614415 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:14.614470 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:14.616930 (+ 2460us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.32.3:44089: BlockingRecv error: recv got EOF from 127.31.32.3:44089 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
Aug 10 04:07:15 dist-test-slave-8w9v krb5kdc[8265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.32.1@KRBTEST.COM for oryx/127.31.32.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 04:07:15 dist-test-slave-8w9v krb5kdc[8265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.32.1@KRBTEST.COM for oryx/127.31.32.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 04:07:15.381798 9387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:15.376943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:43401 (local address 127.31.32.1:59985)
0810 04:07:15.377121 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0810 04:07:15.377132 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 04:07:15.377186 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:07:15.377351 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:07:15.377351 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:07:15.377513 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:07:15.377564 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:15.377566 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:15.378212 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:15.378213 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:15.378717 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:15.378719 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:15.379042 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:15.379043 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:15.379083 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:15.379621 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:07:15.379633 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:07:15.381690 (+ 2057us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.32.62:43401: Server oryx/127.31.32.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":94,"threads_started":1}
W20260810 04:07:15.381850 9389 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:07:15.377033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.1:59985 (local address 127.31.32.62:43401)
0810 04:07:15.377171 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0810 04:07:15.377173 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:07:15.377182 (+ 9us) server_negotiation.cc:407] Connection header received
0810 04:07:15.377233 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:07:15.377234 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:07:15.377252 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:07:15.377316 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:07:15.377659 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:15.378145 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:15.378791 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:15.378979 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:15.379031 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:15.381779 (+ 2748us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.32.1:59985: BlockingRecv error: recv got EOF from 127.31.32.1:59985 (error 108)
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":57,"threads_started":1}
Aug 10 04:07:15 dist-test-slave-8w9v krb5kdc[8265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.32.2@KRBTEST.COM for oryx/127.31.32.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 04:07:15 dist-test-slave-8w9v krb5kdc[8265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.32.2@KRBTEST.COM for oryx/127.31.32.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 04:07:15.503633 9390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:15.498778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:43401 (local address 127.31.32.2:60189)
0810 04:07:15.498931 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0810 04:07:15.498942 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 04:07:15.499009 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:07:15.499121 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:07:15.499122 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:07:15.499346 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:07:15.499395 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:15.499397 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:15.500056 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:15.500057 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:15.500625 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:15.500627 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:15.500973 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:15.500974 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:15.501023 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:15.501550 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:07:15.501555 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:07:15.503572 (+ 2017us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.32.62:43401: Server oryx/127.31.32.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":75,"threads_started":1}
W20260810 04:07:15.503799 9389 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:07:15.498820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.2:60189 (local address 127.31.32.62:43401)
0810 04:07:15.498868 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0810 04:07:15.498871 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:07:15.499012 (+ 141us) server_negotiation.cc:407] Connection header received
0810 04:07:15.499062 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:07:15.499063 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:07:15.499075 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:07:15.499114 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:07:15.499478 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:15.499982 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:15.500697 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:15.500904 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:15.500959 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:15.503693 (+ 2734us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.32.2:60189: BlockingRecv error: recv got EOF from 127.31.32.2:60189 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
Aug 10 04:07:15 dist-test-slave-8w9v krb5kdc[8265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.32.3@KRBTEST.COM for oryx/127.31.32.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 04:07:15 dist-test-slave-8w9v krb5kdc[8265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.32.3@KRBTEST.COM for oryx/127.31.32.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 04:07:15.622071 9391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:15.617401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:43401 (local address 127.31.32.3:51843)
0810 04:07:15.617575 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0810 04:07:15.617585 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 04:07:15.617652 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:07:15.617813 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:07:15.617814 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:07:15.618000 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:07:15.618048 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:15.618051 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:15.618724 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:15.618725 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:15.619254 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:15.619255 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:15.619618 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:15.619619 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:15.619678 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:15.620163 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:07:15.620168 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:07:15.621975 (+ 1807us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.32.62:43401: Server oryx/127.31.32.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":99,"threads_started":1}
W20260810 04:07:15.622129 9389 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:07:15.617463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.3:51843 (local address 127.31.32.62:43401)
0810 04:07:15.617572 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0810 04:07:15.617574 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:07:15.617642 (+ 68us) server_negotiation.cc:407] Connection header received
0810 04:07:15.617713 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:07:15.617714 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:07:15.617729 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:07:15.617800 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:07:15.618128 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:15.618629 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:15.619357 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:15.619543 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:15.619618 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:15.622065 (+ 2447us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.32.3:51843: BlockingRecv error: recv got EOF from 127.31.32.3:51843 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Aug 10 04:07:16 dist-test-slave-8w9v krb5kdc[8265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.32.1@KRBTEST.COM for oryx/127.31.32.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 04:07:16 dist-test-slave-8w9v krb5kdc[8265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.32.1@KRBTEST.COM for oryx/127.31.32.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 04:07:16.386652 9392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:16.382247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:43401 (local address 127.31.32.1:40477)
0810 04:07:16.382396 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0810 04:07:16.382408 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 04:07:16.382475 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:07:16.382623 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:07:16.382624 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:07:16.382779 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:07:16.382841 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:16.382843 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:16.383501 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:16.383503 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:16.384013 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:16.384014 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:16.384397 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:16.384398 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:16.384451 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:16.384873 (+ 422us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:07:16.384878 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:07:16.386595 (+ 1717us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.32.62:43401: Server oryx/127.31.32.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":92,"threads_started":1}
W20260810 04:07:16.386807 9393 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:07:16.382324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.1:40477 (local address 127.31.32.62:43401)
0810 04:07:16.382439 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0810 04:07:16.382441 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:07:16.382475 (+ 34us) server_negotiation.cc:407] Connection header received
0810 04:07:16.382534 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:07:16.382535 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:07:16.382545 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:07:16.382604 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:07:16.382901 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:16.383428 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:16.384083 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:16.384325 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:16.384388 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:16.386710 (+ 2322us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.32.1:40477: BlockingRecv error: recv got EOF from 127.31.32.1:40477 (error 108)
Metrics: {"server-negotiator.queue_time_us":94,"thread_start_us":58,"threads_started":1}
Aug 10 04:07:16 dist-test-slave-8w9v krb5kdc[8265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.32.2@KRBTEST.COM for oryx/127.31.32.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 04:07:16 dist-test-slave-8w9v krb5kdc[8265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.32.2@KRBTEST.COM for oryx/127.31.32.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 04:07:16.512439 9394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:16.504240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:43401 (local address 127.31.32.2:53053)
0810 04:07:16.504413 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0810 04:07:16.504424 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 04:07:16.504493 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:07:16.504635 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:07:16.504635 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:07:16.504816 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:07:16.504870 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:16.504871 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:16.505604 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:16.505605 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:16.506144 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:16.506146 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:16.506521 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:16.506522 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:16.506564 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:16.510410 (+ 3846us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:07:16.510416 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:07:16.512378 (+ 1962us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.32.62:43401: Server oryx/127.31.32.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":95,"threads_started":1}
W20260810 04:07:16.512562 9393 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:07:16.504366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.2:53053 (local address 127.31.32.62:43401)
0810 04:07:16.504446 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0810 04:07:16.504448 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:07:16.504473 (+ 25us) server_negotiation.cc:407] Connection header received
0810 04:07:16.504553 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:07:16.504554 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:07:16.504565 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:07:16.504613 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:07:16.504975 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:16.505518 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:16.506233 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:16.506448 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:16.506503 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:16.512466 (+ 5963us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.32.2:53053: BlockingRecv error: recv got EOF from 127.31.32.2:53053 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Aug 10 04:07:16 dist-test-slave-8w9v krb5kdc[8265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.32.3@KRBTEST.COM for oryx/127.31.32.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 04:07:16 dist-test-slave-8w9v krb5kdc[8265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.32.3@KRBTEST.COM for oryx/127.31.32.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 04:07:16.627702 9395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:16.622617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:43401 (local address 127.31.32.3:53079)
0810 04:07:16.623138 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0810 04:07:16.623156 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 04:07:16.623220 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:07:16.623401 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:07:16.623401 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:07:16.623584 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:07:16.623638 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:16.623640 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:16.624292 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:16.624293 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:16.624814 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:16.624816 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:16.625141 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:16.625142 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:16.625186 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:16.625760 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:07:16.625765 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:07:16.627644 (+ 1879us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.32.62:43401: Server oryx/127.31.32.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":464,"threads_started":1}
W20260810 04:07:16.627838 9393 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:07:16.622729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.3:53079 (local address 127.31.32.62:43401)
0810 04:07:16.622799 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0810 04:07:16.622801 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:07:16.623223 (+ 422us) server_negotiation.cc:407] Connection header received
0810 04:07:16.623289 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:07:16.623289 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:07:16.623321 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:07:16.623372 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:07:16.623705 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:16.624197 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:16.624891 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:16.625068 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:16.625113 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:16.627758 (+ 2645us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.32.3:53079: BlockingRecv error: recv got EOF from 127.31.32.3:53079 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Aug 10 04:07:17 dist-test-slave-8w9v krb5kdc[8265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.32.1@KRBTEST.COM for oryx/127.31.32.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 04:07:17 dist-test-slave-8w9v krb5kdc[8265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.32.1@KRBTEST.COM for oryx/127.31.32.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 04:07:17.391814 9396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:17.387103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:43401 (local address 127.31.32.1:56483)
0810 04:07:17.387259 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0810 04:07:17.387270 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 04:07:17.387343 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:07:17.387476 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:07:17.387477 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:07:17.387649 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:07:17.387700 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:17.387702 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:17.388375 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:17.388376 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:17.388931 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:17.388933 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:17.389251 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:17.389252 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:17.389291 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:17.389825 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:07:17.389831 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:07:17.391745 (+ 1914us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.32.62:43401: Server oryx/127.31.32.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":87,"threads_started":1}
W20260810 04:07:17.391916 9397 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:07:17.387189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.1:56483 (local address 127.31.32.62:43401)
0810 04:07:17.387337 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0810 04:07:17.387339 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:07:17.387348 (+ 9us) server_negotiation.cc:407] Connection header received
0810 04:07:17.387389 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:07:17.387389 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:07:17.387399 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:07:17.387456 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:07:17.387761 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:17.388301 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:17.389000 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:17.389189 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:17.389239 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:17.391825 (+ 2586us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.32.1:56483: BlockingRecv error: recv got EOF from 127.31.32.1:56483 (error 108)
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":88,"threads_started":1}
Aug 10 04:07:17 dist-test-slave-8w9v krb5kdc[8265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.32.2@KRBTEST.COM for oryx/127.31.32.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 04:07:17 dist-test-slave-8w9v krb5kdc[8265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.32.2@KRBTEST.COM for oryx/127.31.32.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 04:07:17.517741 9398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:17.512851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:43401 (local address 127.31.32.2:37045)
0810 04:07:17.513022 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0810 04:07:17.513035 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 04:07:17.513142 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:07:17.513311 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:07:17.513312 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:07:17.513495 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:07:17.513543 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:17.513545 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:17.514265 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:17.514266 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:17.514909 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:17.514911 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:17.515314 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:17.515315 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:17.515367 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:17.515849 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:07:17.515854 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:07:17.517674 (+ 1820us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.32.62:43401: Server oryx/127.31.32.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":96,"threads_started":1}
W20260810 04:07:17.517869 9397 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:07:17.512965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.2:37045 (local address 127.31.32.62:43401)
0810 04:07:17.513036 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0810 04:07:17.513038 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:07:17.513123 (+ 85us) server_negotiation.cc:407] Connection header received
0810 04:07:17.513219 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:07:17.513220 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:07:17.513232 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:07:17.513306 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:07:17.513632 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:17.514185 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:17.514988 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:17.515237 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:17.515323 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:17.517758 (+ 2435us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.32.2:37045: BlockingRecv error: recv got EOF from 127.31.32.2:37045 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Aug 10 04:07:17 dist-test-slave-8w9v krb5kdc[8265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.32.3@KRBTEST.COM for oryx/127.31.32.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 04:07:17 dist-test-slave-8w9v krb5kdc[8265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.32.3@KRBTEST.COM for oryx/127.31.32.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 04:07:17.632678 9399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:17.628217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:43401 (local address 127.31.32.3:35815)
0810 04:07:17.628369 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0810 04:07:17.628388 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 04:07:17.628448 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:07:17.628589 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:07:17.628590 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:07:17.628752 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:07:17.628801 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:17.628803 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:17.629458 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:17.629459 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:17.629977 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:17.629978 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:17.630294 (+ 316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:17.630294 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:17.630340 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:17.630800 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:07:17.630804 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:07:17.632612 (+ 1808us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.32.62:43401: Server oryx/127.31.32.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":71,"threads_started":1}
W20260810 04:07:17.632933 9397 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:07:17.628274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.3:35815 (local address 127.31.32.62:43401)
0810 04:07:17.628345 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0810 04:07:17.628347 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:07:17.628451 (+ 104us) server_negotiation.cc:407] Connection header received
0810 04:07:17.628502 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:07:17.628503 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:07:17.628515 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:07:17.628557 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:07:17.628862 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:17.629392 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:17.630045 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:17.630215 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:17.630280 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:17.632886 (+ 2606us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.32.3:35815: BlockingRecv error: recv got EOF from 127.31.32.3:35815 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Aug 10 04:07:18 dist-test-slave-8w9v krb5kdc[8265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.32.1@KRBTEST.COM for oryx/127.31.32.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 04:07:18 dist-test-slave-8w9v krb5kdc[8265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.32.1@KRBTEST.COM for oryx/127.31.32.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 04:07:18.397267 9400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:18.392470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:43401 (local address 127.31.32.1:44971)
0810 04:07:18.392663 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0810 04:07:18.392676 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 04:07:18.392765 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:07:18.393001 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:07:18.393002 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:07:18.393268 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:07:18.393351 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:18.393354 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:18.394003 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:18.394004 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:18.394527 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:18.394528 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:18.394852 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:18.394853 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:18.394889 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:18.395339 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:07:18.395344 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:07:18.397211 (+ 1867us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.32.62:43401: Server oryx/127.31.32.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":107,"threads_started":1}
W20260810 04:07:18.397476 9401 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:07:18.392563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.1:44971 (local address 127.31.32.62:43401)
0810 04:07:18.392703 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0810 04:07:18.392705 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:07:18.392743 (+ 38us) server_negotiation.cc:407] Connection header received
0810 04:07:18.392866 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:07:18.392866 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:07:18.392876 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:07:18.392972 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:07:18.393437 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:18.393936 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:18.394596 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:18.394779 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:18.394839 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:18.397413 (+ 2574us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.32.1:44971: BlockingRecv error: recv got EOF from 127.31.32.1:44971 (error 108)
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":62,"threads_started":1}
Aug 10 04:07:18 dist-test-slave-8w9v krb5kdc[8265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.32.2@KRBTEST.COM for oryx/127.31.32.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 04:07:18 dist-test-slave-8w9v krb5kdc[8265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.32.2@KRBTEST.COM for oryx/127.31.32.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 04:07:18.523571 9402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:18.518391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:43401 (local address 127.31.32.2:38103)
0810 04:07:18.518580 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0810 04:07:18.518590 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 04:07:18.518647 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:07:18.518829 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:07:18.518830 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:07:18.519006 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:07:18.519058 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:18.519059 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:18.519727 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:18.519728 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:18.520320 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:18.520322 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:18.520660 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:18.520661 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:18.520702 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:18.521164 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:07:18.521169 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:07:18.523500 (+ 2331us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.32.62:43401: Server oryx/127.31.32.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":102,"threads_started":1}
W20260810 04:07:18.523749 9401 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:07:18.518482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.2:38103 (local address 127.31.32.62:43401)
0810 04:07:18.518573 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0810 04:07:18.518575 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:07:18.518648 (+ 73us) server_negotiation.cc:407] Connection header received
0810 04:07:18.518714 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:07:18.518715 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:07:18.518727 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:07:18.518773 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:07:18.519135 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:18.519661 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:18.520407 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:18.520594 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:18.520650 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:18.523627 (+ 2977us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.32.2:38103: BlockingRecv error: recv got EOF from 127.31.32.2:38103 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Aug 10 04:07:18 dist-test-slave-8w9v krb5kdc[8265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.32.3@KRBTEST.COM for oryx/127.31.32.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 04:07:18 dist-test-slave-8w9v krb5kdc[8265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.32.3@KRBTEST.COM for oryx/127.31.32.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 04:07:18.638236 9403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:18.633218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:43401 (local address 127.31.32.3:46615)
0810 04:07:18.633365 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0810 04:07:18.633386 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 04:07:18.633459 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:07:18.633609 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:07:18.633610 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:07:18.633774 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:07:18.633823 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:18.633825 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:18.634433 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:18.634433 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:18.634947 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:18.634948 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:18.635265 (+ 317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:18.635266 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:18.635305 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:18.635741 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:07:18.635745 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:07:18.638158 (+ 2413us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.32.62:43401: Server oryx/127.31.32.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":91,"threads_started":1}
W20260810 04:07:18.638420 9401 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:07:18.633328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.3:46615 (local address 127.31.32.62:43401)
0810 04:07:18.633387 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0810 04:07:18.633389 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:07:18.633446 (+ 57us) server_negotiation.cc:407] Connection header received
0810 04:07:18.633523 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:07:18.633523 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:07:18.633536 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:07:18.633589 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:07:18.633884 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:18.634368 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:18.635027 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:18.635204 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:18.635250 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:18.638333 (+ 3083us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.32.3:46615: BlockingRecv error: recv got EOF from 127.31.32.3:46615 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Aug 10 04:07:19 dist-test-slave-8w9v krb5kdc[8265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.32.1@KRBTEST.COM for oryx/127.31.32.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 04:07:19 dist-test-slave-8w9v krb5kdc[8265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.32.1@KRBTEST.COM for oryx/127.31.32.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 04:07:19.402406 9404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:19.397798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:43401 (local address 127.31.32.1:41547)
0810 04:07:19.397953 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0810 04:07:19.397963 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 04:07:19.398018 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:07:19.398142 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:07:19.398143 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:07:19.398301 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:07:19.398349 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:19.398351 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:19.398934 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:19.398935 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:19.399436 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:19.399438 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:19.399837 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:19.399838 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:19.399919 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:19.400551 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:07:19.400558 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:07:19.402324 (+ 1766us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.32.62:43401: Server oryx/127.31.32.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":107,"threads_started":1}
W20260810 04:07:19.402487 9405 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:07:19.397875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.1:41547 (local address 127.31.32.62:43401)
0810 04:07:19.397980 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0810 04:07:19.397982 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:07:19.398015 (+ 33us) server_negotiation.cc:407] Connection header received
0810 04:07:19.398075 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:07:19.398076 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:07:19.398086 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:07:19.398138 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:07:19.398412 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:19.398872 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:19.399501 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:19.399749 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:19.399814 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:19.402419 (+ 2605us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.32.1:41547: BlockingRecv error: recv got EOF from 127.31.32.1:41547 (error 108)
Metrics: {"server-negotiator.queue_time_us":84,"thread_start_us":50,"threads_started":1}
Aug 10 04:07:19 dist-test-slave-8w9v krb5kdc[8265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.32.2@KRBTEST.COM for oryx/127.31.32.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 04:07:19 dist-test-slave-8w9v krb5kdc[8265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.32.2@KRBTEST.COM for oryx/127.31.32.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 04:07:19.528658 9406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:19.524203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:43401 (local address 127.31.32.2:43589)
0810 04:07:19.524346 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0810 04:07:19.524364 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 04:07:19.524422 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:07:19.524549 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:07:19.524549 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:07:19.524714 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:07:19.524766 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:19.524768 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:19.525375 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:19.525376 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:19.525907 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:19.525908 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:19.526218 (+ 310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:19.526218 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:19.526257 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:19.526682 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:07:19.526686 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:07:19.528561 (+ 1875us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.32.62:43401: Server oryx/127.31.32.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":87,"threads_started":1}
W20260810 04:07:19.528707 9405 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:07:19.524255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.2:43589 (local address 127.31.32.62:43401)
0810 04:07:19.524326 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0810 04:07:19.524327 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 04:07:19.524408 (+ 81us) server_negotiation.cc:407] Connection header received
0810 04:07:19.524467 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:07:19.524468 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:07:19.524478 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:07:19.524530 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:07:19.524837 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:19.525302 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:19.525985 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:19.526158 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:19.526206 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:19.528642 (+ 2436us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.32.2:43589: BlockingRecv error: recv got EOF from 127.31.32.2:43589 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Aug 10 04:07:19 dist-test-slave-8w9v krb5kdc[8265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.32.3@KRBTEST.COM for oryx/127.31.32.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 04:07:19 dist-test-slave-8w9v krb5kdc[8265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.32.3@KRBTEST.COM for oryx/127.31.32.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 04:07:19.645061 9407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:19.639168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:43401 (local address 127.31.32.3:37397)
0810 04:07:19.639311 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0810 04:07:19.639323 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 04:07:19.639452 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:07:19.639612 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:07:19.639613 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:07:19.639782 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:07:19.639836 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:19.639838 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:19.640561 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:19.640562 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:19.641255 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:19.641258 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:19.641711 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:19.641713 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:19.641777 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:19.642404 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:07:19.642409 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:07:19.644962 (+ 2553us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.32.62:43401: Server oryx/127.31.32.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":89,"threads_started":1}
W20260810 04:07:19.645104 9405 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:07:19.639233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.3:37397 (local address 127.31.32.62:43401)
0810 04:07:19.639302 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0810 04:07:19.639304 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:07:19.639426 (+ 122us) server_negotiation.cc:407] Connection header received
0810 04:07:19.639507 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:07:19.639508 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:07:19.639520 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:07:19.639579 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:07:19.639902 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:19.640495 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:19.641371 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:19.641637 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:19.641710 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:19.645041 (+ 3331us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.32.3:37397: BlockingRecv error: recv got EOF from 127.31.32.3:37397 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Aug 10 04:07:20 dist-test-slave-8w9v krb5kdc[8265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.32.1@KRBTEST.COM for oryx/127.31.32.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 04:07:20 dist-test-slave-8w9v krb5kdc[8265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.32.1@KRBTEST.COM for oryx/127.31.32.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 04:07:20.407718 9409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:20.402947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:43401 (local address 127.31.32.1:38907)
0810 04:07:20.403109 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0810 04:07:20.403121 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 04:07:20.403174 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:07:20.403422 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:07:20.403422 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:07:20.403595 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:07:20.403648 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:20.403650 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:20.404385 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:20.404386 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:20.404906 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:20.404908 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:20.405240 (+ 332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:20.405241 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:20.405283 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:20.405785 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:07:20.405790 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:07:20.407656 (+ 1866us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.32.62:43401: Server oryx/127.31.32.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":96,"threads_started":1}
W20260810 04:07:20.407876 9410 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:07:20.403106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.1:38907 (local address 127.31.32.62:43401)
0810 04:07:20.403257 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0810 04:07:20.403259 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:07:20.403267 (+ 8us) server_negotiation.cc:407] Connection header received
0810 04:07:20.403309 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:07:20.403310 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:07:20.403321 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:07:20.403394 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:07:20.403727 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:20.404314 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:20.404978 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:20.405173 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:20.405223 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:20.407783 (+ 2560us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.32.1:38907: BlockingRecv error: recv got EOF from 127.31.32.1:38907 (error 108)
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":83,"threads_started":1}
Aug 10 04:07:20 dist-test-slave-8w9v krb5kdc[8265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.32.2@KRBTEST.COM for oryx/127.31.32.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 04:07:20 dist-test-slave-8w9v krb5kdc[8265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.32.2@KRBTEST.COM for oryx/127.31.32.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 04:07:20.533839 9411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:20.529114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:43401 (local address 127.31.32.2:39329)
0810 04:07:20.529272 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0810 04:07:20.529282 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 04:07:20.529337 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:07:20.529447 (+ 110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:07:20.529447 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:07:20.529608 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:07:20.529661 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:20.529662 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:20.530535 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:20.530536 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:20.531069 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:20.531071 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:20.531423 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:20.531424 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:20.531466 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:20.531918 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:07:20.531923 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:07:20.533781 (+ 1858us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.32.62:43401: Server oryx/127.31.32.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":98,"threads_started":1}
W20260810 04:07:20.533987 9410 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:07:20.529200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.2:39329 (local address 127.31.32.62:43401)
0810 04:07:20.529261 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0810 04:07:20.529263 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:07:20.529349 (+ 86us) server_negotiation.cc:407] Connection header received
0810 04:07:20.529378 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:07:20.529378 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:07:20.529388 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:07:20.529443 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:07:20.529755 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:20.530444 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:20.531150 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:20.531353 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:20.531412 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:20.533904 (+ 2492us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.32.2:39329: BlockingRecv error: recv got EOF from 127.31.32.2:39329 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Aug 10 04:07:20 dist-test-slave-8w9v krb5kdc[8265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.32.3@KRBTEST.COM for oryx/127.31.32.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 04:07:20 dist-test-slave-8w9v krb5kdc[8265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.32.3@KRBTEST.COM for oryx/127.31.32.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 04:07:20.651147 9412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:20.645579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:43401 (local address 127.31.32.3:35553)
0810 04:07:20.645724 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0810 04:07:20.645735 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 04:07:20.645816 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:07:20.645990 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:07:20.645992 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:07:20.646238 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:07:20.646316 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:20.646318 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:20.647266 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:20.647267 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:20.648264 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:20.648291 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:20.648627 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:20.648628 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:20.648671 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:20.649079 (+ 408us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:07:20.649083 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:07:20.651040 (+ 1957us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.32.62:43401: Server oryx/127.31.32.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":88,"threads_started":1}
W20260810 04:07:20.651190 9410 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:07:20.645637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.3:35553 (local address 127.31.32.62:43401)
0810 04:07:20.645705 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0810 04:07:20.645707 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:07:20.645796 (+ 89us) server_negotiation.cc:407] Connection header received
0810 04:07:20.645876 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:07:20.645877 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:07:20.645892 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:07:20.645964 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:07:20.646443 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:20.647170 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:20.648365 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:20.648562 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:20.648618 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:20.651129 (+ 2511us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.32.3:35553: BlockingRecv error: recv got EOF from 127.31.32.3:35553 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Aug 10 04:07:21 dist-test-slave-8w9v krb5kdc[8265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.32.1@KRBTEST.COM for oryx/127.31.32.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 04:07:21 dist-test-slave-8w9v krb5kdc[8265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.32.1@KRBTEST.COM for oryx/127.31.32.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 04:07:21.413112 9413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:21.408321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:43401 (local address 127.31.32.1:57007)
0810 04:07:21.408460 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0810 04:07:21.408471 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 04:07:21.408529 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:07:21.408742 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:07:21.408743 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:07:21.408929 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:07:21.408980 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:21.408982 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:21.409635 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:21.409635 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:21.410162 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:21.410164 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:21.410507 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:21.410508 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:21.410549 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:21.411086 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:07:21.411091 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:07:21.413019 (+ 1928us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.32.62:43401: Server oryx/127.31.32.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":74,"threads_started":1}
W20260810 04:07:21.413166 9414 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:07:21.408349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.1:57007 (local address 127.31.32.62:43401)
0810 04:07:21.408481 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0810 04:07:21.408483 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:07:21.408532 (+ 49us) server_negotiation.cc:407] Connection header received
0810 04:07:21.408585 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:07:21.408586 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:07:21.408596 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:07:21.408663 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:07:21.409058 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:21.409563 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:21.410236 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:21.410440 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:21.410493 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:21.413095 (+ 2602us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.32.1:57007: BlockingRecv error: recv got EOF from 127.31.32.1:57007 (error 108)
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":71,"threads_started":1}
Aug 10 04:07:21 dist-test-slave-8w9v krb5kdc[8265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.32.2@KRBTEST.COM for oryx/127.31.32.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 04:07:21 dist-test-slave-8w9v krb5kdc[8265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.32.2@KRBTEST.COM for oryx/127.31.32.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 04:07:21.543133 9416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:21.534293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:43401 (local address 127.31.32.2:39439)
0810 04:07:21.534488 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0810 04:07:21.534501 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 04:07:21.534576 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:07:21.534742 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:07:21.534742 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:07:21.534936 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:07:21.534985 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:21.534987 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:21.535668 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:21.535669 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:21.536217 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:21.536219 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:21.536591 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:21.536592 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:21.536648 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:21.541209 (+ 4561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:07:21.541214 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:07:21.543031 (+ 1817us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.32.62:43401: Server oryx/127.31.32.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":118,"threads_started":1}
W20260810 04:07:21.543170 9414 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:07:21.534384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.2:39439 (local address 127.31.32.62:43401)
0810 04:07:21.534474 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0810 04:07:21.534477 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:07:21.534568 (+ 91us) server_negotiation.cc:407] Connection header received
0810 04:07:21.534645 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:07:21.534647 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:07:21.534660 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:07:21.534735 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:07:21.535069 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:21.535581 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:21.536305 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:21.536502 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:21.536588 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:21.543111 (+ 6523us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.32.2:39439: BlockingRecv error: recv got EOF from 127.31.32.2:39439 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Aug 10 04:07:21 dist-test-slave-8w9v krb5kdc[8265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.32.3@KRBTEST.COM for oryx/127.31.32.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 04:07:21 dist-test-slave-8w9v krb5kdc[8265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.32.3@KRBTEST.COM for oryx/127.31.32.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 04:07:21.656315 9418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:21.651657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:43401 (local address 127.31.32.3:41239)
0810 04:07:21.651805 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0810 04:07:21.651815 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 04:07:21.651891 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:07:21.652032 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:07:21.652033 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:07:21.652265 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:07:21.652317 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:21.652319 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:21.652923 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:21.652924 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:21.653431 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:21.653432 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:21.653785 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:21.653785 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:21.653826 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:21.654256 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:07:21.654260 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:07:21.656237 (+ 1977us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.32.62:43401: Server oryx/127.31.32.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":102,"threads_started":1}
W20260810 04:07:21.656423 9414 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:07:21.651705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.3:41239 (local address 127.31.32.62:43401)
0810 04:07:21.651770 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0810 04:07:21.651773 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:07:21.651881 (+ 108us) server_negotiation.cc:407] Connection header received
0810 04:07:21.651948 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:07:21.651949 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:07:21.651963 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:07:21.652026 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:07:21.652381 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:21.652851 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:21.653509 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:21.653705 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:21.653752 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:21.656326 (+ 2574us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.32.3:41239: BlockingRecv error: recv got EOF from 127.31.32.3:41239 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Aug 10 04:07:22 dist-test-slave-8w9v krb5kdc[8265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.32.1@KRBTEST.COM for oryx/127.31.32.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 04:07:22 dist-test-slave-8w9v krb5kdc[8265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.32.1@KRBTEST.COM for oryx/127.31.32.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 04:07:22.418692 9419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:22.413680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:43401 (local address 127.31.32.1:37887)
0810 04:07:22.413885 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0810 04:07:22.413897 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 04:07:22.413974 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:07:22.414176 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:07:22.414177 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:07:22.414401 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:07:22.414471 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:22.414473 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:22.415199 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:22.415201 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:22.415783 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:22.415786 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:22.416165 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:22.416166 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:22.416237 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:22.416711 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:07:22.416716 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:07:22.418630 (+ 1914us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.32.62:43401: Server oryx/127.31.32.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":137,"threads_started":1}
W20260810 04:07:22.418805 9420 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:07:22.413772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.1:37887 (local address 127.31.32.62:43401)
0810 04:07:22.413908 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0810 04:07:22.413910 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:07:22.413959 (+ 49us) server_negotiation.cc:407] Connection header received
0810 04:07:22.414036 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:07:22.414036 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:07:22.414090 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:07:22.414166 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:07:22.414587 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:22.415093 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:22.415861 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:22.416062 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:22.416161 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:22.418724 (+ 2563us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.32.1:37887: BlockingRecv error: recv got EOF from 127.31.32.1:37887 (error 108)
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":77,"threads_started":1}
Aug 10 04:07:22 dist-test-slave-8w9v krb5kdc[8265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.32.2@KRBTEST.COM for oryx/127.31.32.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 04:07:22 dist-test-slave-8w9v krb5kdc[8265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.32.2@KRBTEST.COM for oryx/127.31.32.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 04:07:22.547955 9422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:22.543673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:43401 (local address 127.31.32.2:56015)
0810 04:07:22.543829 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0810 04:07:22.543838 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 04:07:22.543888 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:07:22.544023 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:07:22.544023 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:07:22.544202 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:07:22.544252 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:22.544253 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:22.544850 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:22.544851 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:22.545349 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:22.545350 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:22.545647 (+ 297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:22.545648 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:22.545685 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:22.546191 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:07:22.546195 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:07:22.547896 (+ 1701us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.32.62:43401: Server oryx/127.31.32.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":75,"threads_started":1}
W20260810 04:07:22.548085 9420 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:07:22.543732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.2:56015 (local address 127.31.32.62:43401)
0810 04:07:22.543805 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0810 04:07:22.543807 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:07:22.543891 (+ 84us) server_negotiation.cc:407] Connection header received
0810 04:07:22.543945 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:07:22.543946 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:07:22.543957 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:07:22.543998 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:07:22.544321 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:22.544787 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:22.545416 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:22.545588 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:22.545634 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:22.548001 (+ 2367us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.32.2:56015: BlockingRecv error: recv got EOF from 127.31.32.2:56015 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Aug 10 04:07:22 dist-test-slave-8w9v krb5kdc[8265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.32.3@KRBTEST.COM for oryx/127.31.32.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 04:07:22 dist-test-slave-8w9v krb5kdc[8265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.32.3@KRBTEST.COM for oryx/127.31.32.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 04:07:22.661404 9423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:22.656750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:43401 (local address 127.31.32.3:40613)
0810 04:07:22.656924 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0810 04:07:22.656935 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 04:07:22.656999 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:07:22.657153 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:07:22.657153 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:07:22.657344 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:07:22.657392 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:22.657394 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:22.658062 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:22.658063 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:22.658621 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:22.658623 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:22.659009 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:22.659010 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:22.659071 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:22.659568 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:07:22.659573 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:07:22.661345 (+ 1772us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.32.62:43401: Server oryx/127.31.32.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":102,"threads_started":1}
W20260810 04:07:22.661537 9420 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:07:22.656840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.3:40613 (local address 127.31.32.62:43401)
0810 04:07:22.656902 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0810 04:07:22.656905 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:07:22.656978 (+ 73us) server_negotiation.cc:407] Connection header received
0810 04:07:22.657075 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:07:22.657075 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:07:22.657090 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:07:22.657145 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:07:22.657472 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:22.657977 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:22.658705 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:22.658920 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:22.659005 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:22.661434 (+ 2429us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.32.3:40613: BlockingRecv error: recv got EOF from 127.31.32.3:40613 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Aug 10 04:07:23 dist-test-slave-8w9v krb5kdc[8265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.32.1@KRBTEST.COM for oryx/127.31.32.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 04:07:23 dist-test-slave-8w9v krb5kdc[8265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.32.1@KRBTEST.COM for oryx/127.31.32.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 04:07:23.424281 9424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:23.419337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:43401 (local address 127.31.32.1:57749)
0810 04:07:23.419500 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0810 04:07:23.419511 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 04:07:23.419567 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:07:23.419718 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:07:23.419719 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:07:23.419898 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:07:23.419948 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:23.419950 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:23.420674 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:23.420675 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:23.421205 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:23.421206 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:23.421697 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:23.421698 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:23.421749 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:23.422321 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:07:23.422327 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:07:23.424222 (+ 1895us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.32.62:43401: Server oryx/127.31.32.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":101,"threads_started":1}
W20260810 04:07:23.424458 9425 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:07:23.419424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.1:57749 (local address 127.31.32.62:43401)
0810 04:07:23.419537 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0810 04:07:23.419539 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:07:23.419551 (+ 12us) server_negotiation.cc:407] Connection header received
0810 04:07:23.419633 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:07:23.419633 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:07:23.419644 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:07:23.419705 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:07:23.420012 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:23.420600 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:23.421364 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:23.421595 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:23.421655 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:23.424324 (+ 2669us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.32.1:57749: BlockingRecv error: recv got EOF from 127.31.32.1:57749 (error 108)
Metrics: {"server-negotiator.queue_time_us":94,"thread_start_us":54,"threads_started":1}
Aug 10 04:07:23 dist-test-slave-8w9v krb5kdc[8265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.32.2@KRBTEST.COM for oryx/127.31.32.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 04:07:23 dist-test-slave-8w9v krb5kdc[8265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.32.2@KRBTEST.COM for oryx/127.31.32.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 04:07:23.553261 9427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:23.548422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:43401 (local address 127.31.32.2:59875)
0810 04:07:23.548570 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0810 04:07:23.548582 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 04:07:23.548640 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:07:23.548794 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:07:23.548795 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:07:23.548976 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:07:23.549026 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:23.549028 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:23.549701 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:23.549702 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:23.550229 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:23.550231 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:23.550570 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:23.550570 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:23.550611 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:23.551101 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:07:23.551105 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:07:23.553161 (+ 2056us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.32.62:43401: Server oryx/127.31.32.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":82,"threads_started":1}
W20260810 04:07:23.553303 9425 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:07:23.548502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.2:59875 (local address 127.31.32.62:43401)
0810 04:07:23.548574 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0810 04:07:23.548577 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:07:23.548636 (+ 59us) server_negotiation.cc:407] Connection header received
0810 04:07:23.548714 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:07:23.548715 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:07:23.548726 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:07:23.548771 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:07:23.549087 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:23.549630 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:23.550300 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:23.550503 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:23.550558 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:23.553239 (+ 2681us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.32.2:59875: BlockingRecv error: recv got EOF from 127.31.32.2:59875 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Aug 10 04:07:23 dist-test-slave-8w9v krb5kdc[8265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.32.3@KRBTEST.COM for oryx/127.31.32.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 04:07:23 dist-test-slave-8w9v krb5kdc[8265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.32.3@KRBTEST.COM for oryx/127.31.32.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 04:07:23.666528 9428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:23.661864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:43401 (local address 127.31.32.3:39013)
0810 04:07:23.662047 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0810 04:07:23.662057 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 04:07:23.662114 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:07:23.662245 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:07:23.662246 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:07:23.662451 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:07:23.662502 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:23.662504 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:23.663114 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:23.663115 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:23.663700 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:23.663701 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:23.664028 (+ 327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:23.664029 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:23.664073 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:23.664557 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:07:23.664561 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:07:23.666430 (+ 1869us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.32.62:43401: Server oryx/127.31.32.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":105,"threads_started":1}
W20260810 04:07:23.666567 9425 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:07:23.661947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.3:39013 (local address 127.31.32.62:43401)
0810 04:07:23.662006 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0810 04:07:23.662008 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:07:23.662108 (+ 100us) server_negotiation.cc:407] Connection header received
0810 04:07:23.662172 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:07:23.662172 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:07:23.662186 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:07:23.662232 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:07:23.662561 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:23.663043 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:23.663771 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:23.663957 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:23.664010 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:23.666506 (+ 2496us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.32.3:39013: BlockingRecv error: recv got EOF from 127.31.32.3:39013 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Aug 10 04:07:24 dist-test-slave-8w9v krb5kdc[8265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.32.1@KRBTEST.COM for oryx/127.31.32.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 04:07:24 dist-test-slave-8w9v krb5kdc[8265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.32.1@KRBTEST.COM for oryx/127.31.32.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 04:07:24.429531 9429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:24.424790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:43401 (local address 127.31.32.1:36469)
0810 04:07:24.424949 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0810 04:07:24.424961 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 04:07:24.425016 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:07:24.425162 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:07:24.425163 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:07:24.425343 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:07:24.425394 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:24.425395 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:24.426023 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:24.426024 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:24.426531 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:24.426533 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:24.426919 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:24.426921 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:24.426968 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:24.427443 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:07:24.427448 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:07:24.429459 (+ 2011us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.32.62:43401: Server oryx/127.31.32.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":86,"threads_started":1}
W20260810 04:07:24.429714 9431 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:07:24.424907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.1:36469 (local address 127.31.32.62:43401)
0810 04:07:24.425041 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0810 04:07:24.425042 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 04:07:24.425050 (+ 8us) server_negotiation.cc:407] Connection header received
0810 04:07:24.425071 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:07:24.425072 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:07:24.425082 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:07:24.425142 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:07:24.425460 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:24.425960 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:24.426633 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:24.426843 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:24.426904 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:24.429606 (+ 2702us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.32.1:36469: BlockingRecv error: recv got EOF from 127.31.32.1:36469 (error 108)
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":69,"threads_started":1}
Aug 10 04:07:24 dist-test-slave-8w9v krb5kdc[8265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.32.2@KRBTEST.COM for oryx/127.31.32.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 04:07:24 dist-test-slave-8w9v krb5kdc[8265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.32.2@KRBTEST.COM for oryx/127.31.32.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 04:07:24.558684 9432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:24.553796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:43401 (local address 127.31.32.2:54183)
0810 04:07:24.553973 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0810 04:07:24.553992 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 04:07:24.554078 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:07:24.554243 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:07:24.554244 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:07:24.554453 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:07:24.554504 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:24.554505 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:24.555197 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:24.555198 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:24.555733 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:24.555736 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:24.556171 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:24.556173 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:24.556242 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:24.556729 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:07:24.556734 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:07:24.558564 (+ 1830us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.32.62:43401: Server oryx/127.31.32.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":119,"threads_started":1}
W20260810 04:07:24.558724 9431 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:07:24.553884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.2:54183 (local address 127.31.32.62:43401)
0810 04:07:24.553942 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0810 04:07:24.553945 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:07:24.554070 (+ 125us) server_negotiation.cc:407] Connection header received
0810 04:07:24.554144 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:07:24.554145 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:07:24.554160 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:07:24.554238 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:07:24.554593 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:24.555107 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:24.555843 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:24.556069 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:24.556168 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:24.558659 (+ 2491us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.32.2:54183: BlockingRecv error: recv got EOF from 127.31.32.2:54183 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
Aug 10 04:07:24 dist-test-slave-8w9v krb5kdc[8265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.32.3@KRBTEST.COM for oryx/127.31.32.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 04:07:24 dist-test-slave-8w9v krb5kdc[8265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.32.3@KRBTEST.COM for oryx/127.31.32.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 04:07:24.671900 9433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:24.667056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:43401 (local address 127.31.32.3:36605)
0810 04:07:24.667241 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0810 04:07:24.667255 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 04:07:24.667333 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:07:24.667466 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:07:24.667467 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:07:24.667668 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:07:24.667724 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:24.667726 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:24.668456 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:24.668457 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:24.669072 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:24.669074 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:24.669414 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:24.669415 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:24.669463 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:24.669947 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:07:24.669953 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:07:24.671825 (+ 1872us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.32.62:43401: Server oryx/127.31.32.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":121,"threads_started":1}
W20260810 04:07:24.672060 9431 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:07:24.667145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.3:36605 (local address 127.31.32.62:43401)
0810 04:07:24.667206 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0810 04:07:24.667207 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 04:07:24.667327 (+ 120us) server_negotiation.cc:407] Connection header received
0810 04:07:24.667393 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:07:24.667394 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:07:24.667406 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:07:24.667459 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:07:24.667796 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:24.668380 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:24.669149 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:24.669343 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:24.669392 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:24.671973 (+ 2581us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.32.3:36605: BlockingRecv error: recv got EOF from 127.31.32.3:36605 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Aug 10 04:07:25 dist-test-slave-8w9v krb5kdc[8265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.32.1@KRBTEST.COM for oryx/127.31.32.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 04:07:25 dist-test-slave-8w9v krb5kdc[8265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.32.1@KRBTEST.COM for oryx/127.31.32.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 04:07:25.434856 9434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:25.430020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:43401 (local address 127.31.32.1:34487)
0810 04:07:25.430176 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0810 04:07:25.430187 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 04:07:25.430241 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:07:25.430386 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:07:25.430387 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:07:25.430567 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:07:25.430617 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:25.430618 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:25.431292 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:25.431294 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:25.431905 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:25.431908 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:25.432253 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:25.432254 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:25.432331 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:25.432784 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:07:25.432789 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:07:25.434781 (+ 1992us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.32.62:43401: Server oryx/127.31.32.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":88,"threads_started":1}
W20260810 04:07:25.434954 9435 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:07:25.430082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.1:34487 (local address 127.31.32.62:43401)
0810 04:07:25.430217 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0810 04:07:25.430219 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:07:25.430229 (+ 10us) server_negotiation.cc:407] Connection header received
0810 04:07:25.430310 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:07:25.430311 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:07:25.430321 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:07:25.430375 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:07:25.430676 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:25.431195 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:25.431981 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:25.432194 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:25.432246 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:25.434882 (+ 2636us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.32.1:34487: BlockingRecv error: recv got EOF from 127.31.32.1:34487 (error 108)
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":82,"threads_started":1}
Aug 10 04:07:25 dist-test-slave-8w9v krb5kdc[8265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.32.2@KRBTEST.COM for oryx/127.31.32.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 04:07:25 dist-test-slave-8w9v krb5kdc[8265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.32.2@KRBTEST.COM for oryx/127.31.32.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 04:07:25.564172 9436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:25.559195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:43401 (local address 127.31.32.2:50073)
0810 04:07:25.559385 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0810 04:07:25.559396 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 04:07:25.559459 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:07:25.559600 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:07:25.559601 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:07:25.559801 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:07:25.559854 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:25.559856 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:25.560674 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:25.560675 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:25.561263 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:25.561265 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:25.561631 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:25.561632 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:25.561682 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:25.562167 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:07:25.562172 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:07:25.564050 (+ 1878us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.32.62:43401: Server oryx/127.31.32.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":123,"threads_started":1}
W20260810 04:07:25.564385 9435 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:07:25.559304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.2:50073 (local address 127.31.32.62:43401)
0810 04:07:25.559351 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0810 04:07:25.559353 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:07:25.559441 (+ 88us) server_negotiation.cc:407] Connection header received
0810 04:07:25.559508 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:07:25.559508 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:07:25.559519 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:07:25.559567 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:07:25.559954 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:25.560588 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:25.561351 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:25.561551 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:25.561614 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:25.564239 (+ 2625us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.32.2:50073: BlockingRecv error: recv got EOF from 127.31.32.2:50073 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
Aug 10 04:07:25 dist-test-slave-8w9v krb5kdc[8265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.32.3@KRBTEST.COM for oryx/127.31.32.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 04:07:25 dist-test-slave-8w9v krb5kdc[8265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.32.3@KRBTEST.COM for oryx/127.31.32.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 04:07:25.677429 9437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:25.672397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:43401 (local address 127.31.32.3:59037)
0810 04:07:25.672566 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0810 04:07:25.672577 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 04:07:25.672663 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:07:25.672826 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:07:25.672827 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:07:25.673015 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:07:25.673063 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:25.673065 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:25.673725 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:25.673726 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:25.674266 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:25.674268 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:25.674744 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:25.674745 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:25.674807 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:25.675306 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:07:25.675311 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:07:25.677370 (+ 2059us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.32.62:43401: Server oryx/127.31.32.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":99,"threads_started":1}
W20260810 04:07:25.677559 9435 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:07:25.672461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.3:59037 (local address 127.31.32.62:43401)
0810 04:07:25.672535 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0810 04:07:25.672538 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:07:25.672640 (+ 102us) server_negotiation.cc:407] Connection header received
0810 04:07:25.672726 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:07:25.672727 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:07:25.672742 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:07:25.672806 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:07:25.673139 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:25.673641 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:25.674415 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:25.674659 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:25.674741 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:25.677456 (+ 2715us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.32.3:59037: BlockingRecv error: recv got EOF from 127.31.32.3:59037 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
Aug 10 04:07:26 dist-test-slave-8w9v krb5kdc[8265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.32.1@KRBTEST.COM for oryx/127.31.32.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 04:07:26 dist-test-slave-8w9v krb5kdc[8265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.32.1@KRBTEST.COM for oryx/127.31.32.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 04:07:26.440093 9439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:26.435432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:43401 (local address 127.31.32.1:52103)
0810 04:07:26.435586 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0810 04:07:26.435605 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 04:07:26.435656 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:07:26.435798 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:07:26.435799 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:07:26.435979 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:07:26.436040 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:26.436042 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:26.436761 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:26.436762 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:26.437299 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:26.437300 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:26.437679 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:26.437680 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:26.437720 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:26.438187 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:07:26.438191 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:07:26.440023 (+ 1832us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.32.62:43401: Server oryx/127.31.32.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":94,"threads_started":1}
W20260810 04:07:26.440299 9441 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:07:26.435516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.1:52103 (local address 127.31.32.62:43401)
0810 04:07:26.435664 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0810 04:07:26.435667 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:07:26.435683 (+ 16us) server_negotiation.cc:407] Connection header received
0810 04:07:26.435706 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:07:26.435706 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:07:26.435716 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:07:26.435785 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:07:26.436168 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:26.436692 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:26.437399 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:26.437603 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:26.437673 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:26.440178 (+ 2505us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.32.1:52103: BlockingRecv error: recv got EOF from 127.31.32.1:52103 (error 108)
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":91,"threads_started":1}
Aug 10 04:07:26 dist-test-slave-8w9v krb5kdc[8265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.32.2@KRBTEST.COM for oryx/127.31.32.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 04:07:26 dist-test-slave-8w9v krb5kdc[8265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.32.2@KRBTEST.COM for oryx/127.31.32.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 04:07:26.573678 9442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:26.564674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:43401 (local address 127.31.32.2:40345)
0810 04:07:26.564831 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0810 04:07:26.564843 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 04:07:26.564900 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:07:26.565030 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:07:26.565030 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:07:26.565215 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:07:26.565267 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:26.565269 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:26.566050 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:26.566051 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:26.566630 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:26.566632 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:26.566959 (+ 327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:26.566959 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:26.566998 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:26.571394 (+ 4396us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:07:26.571399 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:07:26.573555 (+ 2156us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.32.62:43401: Server oryx/127.31.32.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":101,"threads_started":1}
W20260810 04:07:26.573716 9441 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:07:26.564773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.2:40345 (local address 127.31.32.62:43401)
0810 04:07:26.564823 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0810 04:07:26.564825 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:07:26.564902 (+ 77us) server_negotiation.cc:407] Connection header received
0810 04:07:26.564947 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:07:26.564947 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:07:26.564958 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:07:26.565015 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:07:26.565372 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:26.565971 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:26.566699 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:26.566892 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:26.566990 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:26.573647 (+ 6657us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.32.2:40345: BlockingRecv error: recv got EOF from 127.31.32.2:40345 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
Aug 10 04:07:26 dist-test-slave-8w9v krb5kdc[8265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.32.3@KRBTEST.COM for oryx/127.31.32.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 04:07:26 dist-test-slave-8w9v krb5kdc[8265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.32.3@KRBTEST.COM for oryx/127.31.32.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 04:07:26.682605 9443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:26.677849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:43401 (local address 127.31.32.3:55623)
0810 04:07:26.678050 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0810 04:07:26.678062 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 04:07:26.678157 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:07:26.678306 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:07:26.678307 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:07:26.678494 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:07:26.678541 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:26.678543 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:26.679218 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:26.679219 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:26.679783 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:26.679785 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:26.680243 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:26.680244 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:26.680315 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:26.680723 (+ 408us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:07:26.680728 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:07:26.682496 (+ 1768us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.32.62:43401: Server oryx/127.31.32.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":106,"threads_started":1}
W20260810 04:07:26.682639 9441 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:07:26.677929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.3:55623 (local address 127.31.32.62:43401)
0810 04:07:26.678033 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0810 04:07:26.678036 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:07:26.678134 (+ 98us) server_negotiation.cc:407] Connection header received
0810 04:07:26.678231 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:07:26.678232 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:07:26.678248 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:07:26.678300 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:07:26.678627 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:26.679136 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:26.679871 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:26.680182 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:26.680244 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:26.682573 (+ 2329us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.32.3:55623: BlockingRecv error: recv got EOF from 127.31.32.3:55623 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Aug 10 04:07:27 dist-test-slave-8w9v krb5kdc[8265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.32.1@KRBTEST.COM for oryx/127.31.32.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 04:07:27 dist-test-slave-8w9v krb5kdc[8265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.32.1@KRBTEST.COM for oryx/127.31.32.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 04:07:27.445217 9444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:27.440615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:43401 (local address 127.31.32.1:34743)
0810 04:07:27.440767 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0810 04:07:27.440777 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 04:07:27.440820 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:07:27.440952 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:07:27.440953 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:07:27.441127 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:07:27.441177 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:27.441178 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:27.441840 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:27.441842 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:27.442378 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:27.442380 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:27.442686 (+ 306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:27.442687 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:27.442727 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:27.443254 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:07:27.443259 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:07:27.445121 (+ 1862us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.32.62:43401: Server oryx/127.31.32.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":95,"threads_started":1}
W20260810 04:07:27.445276 9445 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:07:27.440675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.1:34743 (local address 127.31.32.62:43401)
0810 04:07:27.440805 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0810 04:07:27.440806 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 04:07:27.440814 (+ 8us) server_negotiation.cc:407] Connection header received
0810 04:07:27.440865 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:07:27.440866 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:07:27.440875 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:07:27.440948 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:07:27.441257 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:27.441768 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:27.442447 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:27.442627 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:27.442676 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:27.445204 (+ 2528us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.32.1:34743: BlockingRecv error: recv got EOF from 127.31.32.1:34743 (error 108)
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":75,"threads_started":1}
Aug 10 04:07:27 dist-test-slave-8w9v krb5kdc[8265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.32.2@KRBTEST.COM for oryx/127.31.32.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 04:07:27 dist-test-slave-8w9v krb5kdc[8265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.32.2@KRBTEST.COM for oryx/127.31.32.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 04:07:27.579105 9446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:27.574241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:43401 (local address 127.31.32.2:37179)
0810 04:07:27.574406 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0810 04:07:27.574417 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 04:07:27.574478 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:07:27.574591 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:07:27.574592 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:07:27.574756 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:07:27.574806 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:27.574807 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:27.575504 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:27.575506 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:27.576221 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:27.576224 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:27.576575 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:27.576576 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:27.576621 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:27.577201 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:07:27.577207 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:07:27.579045 (+ 1838us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.32.62:43401: Server oryx/127.31.32.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":76,"threads_started":1}
W20260810 04:07:27.579254 9445 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:07:27.574314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.2:37179 (local address 127.31.32.62:43401)
0810 04:07:27.574414 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0810 04:07:27.574416 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:07:27.574472 (+ 56us) server_negotiation.cc:407] Connection header received
0810 04:07:27.574527 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:07:27.574528 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:07:27.574539 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:07:27.574587 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:07:27.574867 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:27.575417 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:27.576303 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:27.576506 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:27.576563 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:27.579172 (+ 2609us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.32.2:37179: BlockingRecv error: recv got EOF from 127.31.32.2:37179 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
I20260810 04:07:27.609148 31872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskws89Io/build/release/bin/kudu with pid 8910
I20260810 04:07:27.616372 31872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskws89Io/build/release/bin/kudu with pid 9080
I20260810 04:07:27.621783 31872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskws89Io/build/release/bin/kudu with pid 9225
I20260810 04:07:27.627482 31872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskws89Io/build/release/bin/kudu with pid 8834
2026-08-10T04:07:27Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17506 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-08-10T04:07:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T04:07:27Z Disabled control of system clock
I20260810 04:07:27.657423 31872 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskws89Io/build/release/bin/kudu
/tmp/dist-test-taskws89Io/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786334794676589-31872-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786334794676589-31872-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786334794676589-31872-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786334794676589-31872-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.32.62:33867
--webserver_interface=127.31.32.62
--webserver_port=0
--builtin_ntp_servers=127.31.32.20:34729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.32.62:33867
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260810 04:07:27.737123 9450 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:07:27.737306 9450 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:07:27.737325 9450 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:07:27.738375 9450 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:07:27.738408 9450 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:07:27.738420 9450 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:07:27.738432 9450 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:07:27.738443 9450 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:07:27.739673 9450 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.32.20:34729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786334794676589-31872-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786334794676589-31872-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.32.62:33867
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.32.62:33867
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786334794676589-31872-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.32.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9450
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786334794676589-31872-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:07:27.739857 9450 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:07:27.740079 9450 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:07:27.745620 9459 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:07:27.745774 9456 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:07:27.745843 9450 server_base.cc:1034] running on GCE node
W20260810 04:07:27.745874 9455 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:07:27.746275 9450 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:07:27.746629 9450 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:07:27.749969 9450 hybrid_clock.cc:648] HybridClock initialized: now 1786334847749938 us; error 36 us; skew 500 ppm
I20260810 04:07:27.751681 9450 webserver.cc:460] Webserver started at http://127.31.32.62:46079/ using document root <none> and password file <none>
I20260810 04:07:27.751991 9450 fs_manager.cc:359] Metadata directory not provided
I20260810 04:07:27.752106 9450 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:07:27.752328 9450 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:07:27.753952 9450 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786334794676589-31872-0/minicluster-data/master-0/data/instance:
uuid: "745401239b4e4f93b4d938e4e4f43a7b"
format_stamp: "Formatted at 2026-08-10 04:07:27 on dist-test-slave-8w9v"
server_key: "b221e2b73cda8762edadc964dada52f6"
server_key_iv: "9fc651f63e892600cd9b96d36b59597c"
server_key_version: "encryptionkey@0"
I20260810 04:07:27.754390 9450 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786334794676589-31872-0/minicluster-data/master-0/wal/instance:
uuid: "745401239b4e4f93b4d938e4e4f43a7b"
format_stamp: "Formatted at 2026-08-10 04:07:27 on dist-test-slave-8w9v"
server_key: "b221e2b73cda8762edadc964dada52f6"
server_key_iv: "9fc651f63e892600cd9b96d36b59597c"
server_key_version: "encryptionkey@0"
I20260810 04:07:27.756116 9450 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260810 04:07:27.757028 9467 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:07:27.757298 9450 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:07:27.757484 9450 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786334794676589-31872-0/minicluster-data/master-0/data,/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786334794676589-31872-0/minicluster-data/master-0/wal
uuid: "745401239b4e4f93b4d938e4e4f43a7b"
format_stamp: "Formatted at 2026-08-10 04:07:27 on dist-test-slave-8w9v"
server_key: "b221e2b73cda8762edadc964dada52f6"
server_key_iv: "9fc651f63e892600cd9b96d36b59597c"
server_key_version: "encryptionkey@0"
I20260810 04:07:27.757642 9450 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786334794676589-31872-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786334794676589-31872-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786334794676589-31872-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:07:27.783437 9450 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:07:27.783776 9450 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:07:27.783946 9450 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:07:27.787873 9450 rpc_server.cc:307] RPC server started. Bound to: 127.31.32.62:33867
I20260810 04:07:27.787940 9520 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.32.62:33867 every 8 connection(s)
I20260810 04:07:27.788311 9450 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786334794676589-31872-0/minicluster-data/master-0/data/info.pb
I20260810 04:07:27.788998 9521 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:07:27.791616 31872 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskws89Io/build/release/bin/kudu as pid 9450
I20260810 04:07:27.791621 9521 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 745401239b4e4f93b4d938e4e4f43a7b: Bootstrap starting.
I20260810 04:07:27.791708 31872 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786334794676589-31872-0/minicluster-data/master-0/wal/instance
I20260810 04:07:27.791934 31872 external_mini_cluster.cc:1428] Setting key 980bc89d16f0ad48c787e34ef0f078dc
I20260810 04:07:27.792363 9521 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 745401239b4e4f93b4d938e4e4f43a7b: Neither blocks nor log segments found. Creating new log.
I20260810 04:07:27.792734 9521 log.cc:824] T 00000000000000000000000000000000 P 745401239b4e4f93b4d938e4e4f43a7b: Log is configured to *not* fsync() on all Append() calls
I20260810 04:07:27.793522 9521 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 745401239b4e4f93b4d938e4e4f43a7b: No bootstrap required, opened a new log
I20260810 04:07:27.795753 9521 raft_consensus.cc:348] T 00000000000000000000000000000000 P 745401239b4e4f93b4d938e4e4f43a7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "745401239b4e4f93b4d938e4e4f43a7b" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 33867 } }
I20260810 04:07:27.795900 9521 raft_consensus.cc:374] T 00000000000000000000000000000000 P 745401239b4e4f93b4d938e4e4f43a7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:07:27.795943 9521 raft_consensus.cc:729] T 00000000000000000000000000000000 P 745401239b4e4f93b4d938e4e4f43a7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 745401239b4e4f93b4d938e4e4f43a7b, State: Initialized, Role: FOLLOWER
I20260810 04:07:27.796069 9521 consensus_queue.cc:260] T 00000000000000000000000000000000 P 745401239b4e4f93b4d938e4e4f43a7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "745401239b4e4f93b4d938e4e4f43a7b" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 33867 } }
I20260810 04:07:27.796193 9521 raft_consensus.cc:388] T 00000000000000000000000000000000 P 745401239b4e4f93b4d938e4e4f43a7b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 04:07:27.796231 9521 raft_consensus.cc:482] T 00000000000000000000000000000000 P 745401239b4e4f93b4d938e4e4f43a7b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 04:07:27.796270 9521 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 745401239b4e4f93b4d938e4e4f43a7b [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:07:27.796918 9521 raft_consensus.cc:504] T 00000000000000000000000000000000 P 745401239b4e4f93b4d938e4e4f43a7b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "745401239b4e4f93b4d938e4e4f43a7b" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 33867 } }
I20260810 04:07:27.797039 9521 leader_election.cc:302] T 00000000000000000000000000000000 P 745401239b4e4f93b4d938e4e4f43a7b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 745401239b4e4f93b4d938e4e4f43a7b; no voters:
I20260810 04:07:27.797251 9521 leader_election.cc:288] T 00000000000000000000000000000000 P 745401239b4e4f93b4d938e4e4f43a7b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 04:07:27.797278 9526 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 745401239b4e4f93b4d938e4e4f43a7b [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:07:27.797400 9526 raft_consensus.cc:686] T 00000000000000000000000000000000 P 745401239b4e4f93b4d938e4e4f43a7b [term 1 LEADER]: Becoming Leader. State: Replica: 745401239b4e4f93b4d938e4e4f43a7b, State: Running, Role: LEADER
I20260810 04:07:27.797483 9526 consensus_queue.cc:237] T 00000000000000000000000000000000 P 745401239b4e4f93b4d938e4e4f43a7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "745401239b4e4f93b4d938e4e4f43a7b" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 33867 } }
I20260810 04:07:27.797546 9521 sys_catalog.cc:564] T 00000000000000000000000000000000 P 745401239b4e4f93b4d938e4e4f43a7b [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:07:27.797789 9527 sys_catalog.cc:455] T 00000000000000000000000000000000 P 745401239b4e4f93b4d938e4e4f43a7b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "745401239b4e4f93b4d938e4e4f43a7b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "745401239b4e4f93b4d938e4e4f43a7b" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 33867 } } }
I20260810 04:07:27.797991 9528 sys_catalog.cc:455] T 00000000000000000000000000000000 P 745401239b4e4f93b4d938e4e4f43a7b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 745401239b4e4f93b4d938e4e4f43a7b. Latest consensus state: current_term: 1 leader_uuid: "745401239b4e4f93b4d938e4e4f43a7b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "745401239b4e4f93b4d938e4e4f43a7b" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 33867 } } }
I20260810 04:07:27.798062 9527 sys_catalog.cc:458] T 00000000000000000000000000000000 P 745401239b4e4f93b4d938e4e4f43a7b [sys.catalog]: This master's current role is: LEADER
I20260810 04:07:27.798372 9524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:27.792474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52758 (local address 127.31.32.62:33867)
0810 04:07:27.792685 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0810 04:07:27.792694 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 04:07:27.792731 (+ 37us) server_negotiation.cc:407] Connection header received
0810 04:07:27.793338 (+ 607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:07:27.793347 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:07:27.793421 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:07:27.793572 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:07:27.793823 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:27.795222 (+ 1399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:27.795923 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:27.796512 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:27.796632 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:27.797357 (+ 725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:07:27.797361 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:07:27.797361 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:07:27.797572 (+ 211us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:07:27.797614 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:07:27.797629 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:07:27.797880 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0810 04:07:27.797913 (+ 33us) server_negotiation.cc:299] Negotiation successful
0810 04:07:27.798000 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":81,"threads_started":1}
I20260810 04:07:27.798089 9528 sys_catalog.cc:458] T 00000000000000000000000000000000 P 745401239b4e4f93b4d938e4e4f43a7b [sys.catalog]: This master's current role is: LEADER
I20260810 04:07:27.798661 9535 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 04:07:27.799197 9535 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 04:07:27.800727 9535 catalog_manager.cc:1347] Generated new cluster ID: a20932217df244b880467035091e5257
I20260810 04:07:27.800786 9535 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 04:07:27.827872 9535 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 04:07:27.828668 9535 catalog_manager.cc:1504] Loading token signing keys...
I20260810 04:07:27.837246 9535 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 745401239b4e4f93b4d938e4e4f43a7b: Generated new TSK 0
I20260810 04:07:27.837423 9535 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 04:07:27.841060 31872 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskws89Io/build/release/bin/kudu
/tmp/dist-test-taskws89Io/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786334794676589-31872-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786334794676589-31872-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786334794676589-31872-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786334794676589-31872-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.32.1:0
--local_ip_for_outbound_sockets=127.31.32.1
--webserver_interface=127.31.32.1
--webserver_port=0
--tserver_master_addrs=127.31.32.62:33867
--builtin_ntp_servers=127.31.32.20:34729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260810 04:07:27.921305 9546 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:07:27.921489 9546 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:07:27.921509 9546 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:07:27.922587 9546 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:07:27.922623 9546 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:07:27.922647 9546 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.32.1
I20260810 04:07:27.923920 9546 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.32.20:34729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786334794676589-31872-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786334794676589-31872-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.32.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786334794676589-31872-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.32.1
--webserver_port=0
--tserver_master_addrs=127.31.32.62:33867
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9546
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.32.1
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786334794676589-31872-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:07:27.924155 9546 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:07:27.924435 9546 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:07:27.929323 9551 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:07:27.929370 9554 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:07:27.929430 9546 server_base.cc:1034] running on GCE node
W20260810 04:07:27.929541 9552 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:07:27.929757 9546 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:07:27.930004 9546 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:07:27.931166 9546 hybrid_clock.cc:648] HybridClock initialized: now 1786334847931163 us; error 33 us; skew 500 ppm
I20260810 04:07:27.932475 9546 webserver.cc:460] Webserver started at http://127.31.32.1:43535/ using document root <none> and password file <none>
I20260810 04:07:27.932716 9546 fs_manager.cc:359] Metadata directory not provided
I20260810 04:07:27.932796 9546 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:07:27.932934 9546 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:07:27.933998 9546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786334794676589-31872-0/minicluster-data/ts-0/data/instance:
uuid: "c0bd7b689bf347fd811b990762777bc6"
format_stamp: "Formatted at 2026-08-10 04:07:27 on dist-test-slave-8w9v"
server_key: "fc728d6eef9b1fa7460e124831d780b0"
server_key_iv: "f7200d8dbd189294d1cc2c6db9a5811f"
server_key_version: "encryptionkey@0"
I20260810 04:07:27.934373 9546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786334794676589-31872-0/minicluster-data/ts-0/wal/instance:
uuid: "c0bd7b689bf347fd811b990762777bc6"
format_stamp: "Formatted at 2026-08-10 04:07:27 on dist-test-slave-8w9v"
server_key: "fc728d6eef9b1fa7460e124831d780b0"
server_key_iv: "f7200d8dbd189294d1cc2c6db9a5811f"
server_key_version: "encryptionkey@0"
I20260810 04:07:27.935755 9546 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 04:07:27.936578 9562 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:07:27.936736 9546 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260810 04:07:27.936844 9546 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786334794676589-31872-0/minicluster-data/ts-0/data,/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786334794676589-31872-0/minicluster-data/ts-0/wal
uuid: "c0bd7b689bf347fd811b990762777bc6"
format_stamp: "Formatted at 2026-08-10 04:07:27 on dist-test-slave-8w9v"
server_key: "fc728d6eef9b1fa7460e124831d780b0"
server_key_iv: "f7200d8dbd189294d1cc2c6db9a5811f"
server_key_version: "encryptionkey@0"
I20260810 04:07:27.936918 9546 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786334794676589-31872-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786334794676589-31872-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786334794676589-31872-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:07:27.948714 9546 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:07:27.948962 9546 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:07:27.949112 9546 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:07:27.949344 9546 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:07:27.949649 9546 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:07:27.949709 9546 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:07:27.949750 9546 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:07:27.949780 9546 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:07:27.955781 9546 rpc_server.cc:307] RPC server started. Bound to: 127.31.32.1:45159
I20260810 04:07:27.955818 9687 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.32.1:45159 every 8 connection(s)
I20260810 04:07:27.956271 9546 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786334794676589-31872-0/minicluster-data/ts-0/data/info.pb
I20260810 04:07:27.961194 9524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:27.957086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.1:60891 (local address 127.31.32.62:33867)
0810 04:07:27.957156 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0810 04:07:27.957158 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:07:27.957358 (+ 200us) server_negotiation.cc:407] Connection header received
0810 04:07:27.957810 (+ 452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:07:27.957812 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:07:27.957822 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:07:27.957884 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:07:27.958561 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:27.959044 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:27.959928 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:27.960130 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:27.960217 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:27.960976 (+ 759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:07:27.960977 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:07:27.960978 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:07:27.960981 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:07:27.960993 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:07:27.960996 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:07:27.961036 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0810 04:07:27.961117 (+ 81us) server_negotiation.cc:299] Negotiation successful
0810 04:07:27.961137 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260810 04:07:27.961426 9690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:27.957044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:33867 (local address 127.31.32.1:60891)
0810 04:07:27.957234 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0810 04:07:27.957271 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0810 04:07:27.957728 (+ 457us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:07:27.957949 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:07:27.957952 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:07:27.957960 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:07:27.958480 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:27.958485 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:27.959135 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:27.959136 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:27.959838 (+ 702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:27.959839 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:27.960230 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:27.960231 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:27.960339 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:27.960852 (+ 513us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 04:07:27.960853 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:07:27.960880 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 04:07:27.960902 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:07:27.961049 (+ 147us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:07:27.961053 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:07:27.961055 (+ 2us) client_negotiation.cc:769] Sending connection context
0810 04:07:27.961108 (+ 53us) client_negotiation.cc:240] Negotiation successful
0810 04:07:27.961215 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":84,"threads_started":1}
I20260810 04:07:27.961964 9688 heartbeater.cc:347] Connected to a master server at 127.31.32.62:33867
I20260810 04:07:27.962065 9688 heartbeater.cc:464] Registering TS with master...
I20260810 04:07:27.962253 9688 heartbeater.cc:511] Master 127.31.32.62:33867 requested a full tablet report, sending...
I20260810 04:07:27.962749 9485 ts_manager.cc:194] Registered new tserver with Master: c0bd7b689bf347fd811b990762777bc6 (127.31.32.1:45159)
I20260810 04:07:27.963397 9485 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.32.1:60891
I20260810 04:07:27.966426 31872 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskws89Io/build/release/bin/kudu as pid 9546
I20260810 04:07:27.966502 31872 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786334794676589-31872-0/minicluster-data/ts-0/wal/instance
I20260810 04:07:27.966660 31872 external_mini_cluster.cc:1428] Setting key d658a744c5b1358d6c2438621bfdaa9a
I20260810 04:07:27.967579 31872 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskws89Io/build/release/bin/kudu
/tmp/dist-test-taskws89Io/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786334794676589-31872-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786334794676589-31872-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786334794676589-31872-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786334794676589-31872-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.32.2:0
--local_ip_for_outbound_sockets=127.31.32.2
--webserver_interface=127.31.32.2
--webserver_port=0
--tserver_master_addrs=127.31.32.62:33867
--builtin_ntp_servers=127.31.32.20:34729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260810 04:07:28.050563 9691 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:07:28.050771 9691 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:07:28.050793 9691 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:07:28.052052 9691 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:07:28.052095 9691 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:07:28.052172 9691 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.32.2
I20260810 04:07:28.053907 9691 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.32.20:34729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786334794676589-31872-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786334794676589-31872-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.32.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786334794676589-31872-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.32.2
--webserver_port=0
--tserver_master_addrs=127.31.32.62:33867
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9691
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.32.2
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786334794676589-31872-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:07:28.054189 9691 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:07:28.054517 9691 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:07:28.060072 9698 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:07:28.060175 9697 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:07:28.060303 9700 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:07:28.060307 9691 server_base.cc:1034] running on GCE node
I20260810 04:07:28.060556 9691 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:07:28.060824 9691 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:07:28.062000 9691 hybrid_clock.cc:648] HybridClock initialized: now 1786334848061983 us; error 44 us; skew 500 ppm
I20260810 04:07:28.063517 9691 webserver.cc:460] Webserver started at http://127.31.32.2:34273/ using document root <none> and password file <none>
I20260810 04:07:28.063805 9691 fs_manager.cc:359] Metadata directory not provided
I20260810 04:07:28.063886 9691 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:07:28.064050 9691 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:07:28.065173 9691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786334794676589-31872-0/minicluster-data/ts-1/data/instance:
uuid: "db5931177a7e4e8288d676a379588c2c"
format_stamp: "Formatted at 2026-08-10 04:07:28 on dist-test-slave-8w9v"
server_key: "93c048c8cbbc7f7f2ed581699a8b3b27"
server_key_iv: "a6a78f7236e8f66f41e17c2c929d3ac0"
server_key_version: "encryptionkey@0"
I20260810 04:07:28.065568 9691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786334794676589-31872-0/minicluster-data/ts-1/wal/instance:
uuid: "db5931177a7e4e8288d676a379588c2c"
format_stamp: "Formatted at 2026-08-10 04:07:28 on dist-test-slave-8w9v"
server_key: "93c048c8cbbc7f7f2ed581699a8b3b27"
server_key_iv: "a6a78f7236e8f66f41e17c2c929d3ac0"
server_key_version: "encryptionkey@0"
I20260810 04:07:28.066931 9691 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 04:07:28.067656 9707 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:07:28.067848 9691 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 04:07:28.067965 9691 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786334794676589-31872-0/minicluster-data/ts-1/data,/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786334794676589-31872-0/minicluster-data/ts-1/wal
uuid: "db5931177a7e4e8288d676a379588c2c"
format_stamp: "Formatted at 2026-08-10 04:07:28 on dist-test-slave-8w9v"
server_key: "93c048c8cbbc7f7f2ed581699a8b3b27"
server_key_iv: "a6a78f7236e8f66f41e17c2c929d3ac0"
server_key_version: "encryptionkey@0"
I20260810 04:07:28.068050 9691 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786334794676589-31872-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786334794676589-31872-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786334794676589-31872-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:07:28.081179 9691 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:07:28.081511 9691 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:07:28.081676 9691 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:07:28.081915 9691 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:07:28.082288 9691 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:07:28.082351 9691 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:07:28.082398 9691 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:07:28.082433 9691 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:07:28.088733 9691 rpc_server.cc:307] RPC server started. Bound to: 127.31.32.2:34079
I20260810 04:07:28.088773 9836 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.32.2:34079 every 8 connection(s)
I20260810 04:07:28.089118 9691 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786334794676589-31872-0/minicluster-data/ts-1/data/info.pb
I20260810 04:07:28.093092 31872 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskws89Io/build/release/bin/kudu as pid 9691
I20260810 04:07:28.093171 31872 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786334794676589-31872-0/minicluster-data/ts-1/wal/instance
I20260810 04:07:28.093402 31872 external_mini_cluster.cc:1428] Setting key b9ea62e2e196555504ffab43b0a1110d
I20260810 04:07:28.094435 31872 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskws89Io/build/release/bin/kudu
/tmp/dist-test-taskws89Io/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786334794676589-31872-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786334794676589-31872-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786334794676589-31872-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786334794676589-31872-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.32.3:0
--local_ip_for_outbound_sockets=127.31.32.3
--webserver_interface=127.31.32.3
--webserver_port=0
--tserver_master_addrs=127.31.32.62:33867
--builtin_ntp_servers=127.31.32.20:34729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260810 04:07:28.094455 9524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:28.089895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.2:35029 (local address 127.31.32.62:33867)
0810 04:07:28.089985 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0810 04:07:28.089988 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:07:28.090186 (+ 198us) server_negotiation.cc:407] Connection header received
0810 04:07:28.090652 (+ 466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:07:28.090653 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:07:28.090665 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:07:28.090731 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:07:28.091410 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:28.091920 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:28.092943 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:28.093125 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:28.093236 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:28.094177 (+ 941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:07:28.094178 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:07:28.094178 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:07:28.094183 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:07:28.094197 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:07:28.094202 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:07:28.094248 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0810 04:07:28.094349 (+ 101us) server_negotiation.cc:299] Negotiation successful
0810 04:07:28.094375 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 04:07:28.094659 9839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:28.089863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:33867 (local address 127.31.32.2:35029)
0810 04:07:28.090065 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0810 04:07:28.090099 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0810 04:07:28.090568 (+ 469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:07:28.090797 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:07:28.090800 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:07:28.090808 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:07:28.091336 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:28.091341 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:28.092005 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:28.092006 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:28.092840 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:28.092842 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:28.093255 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:28.093256 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:28.093415 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:28.094025 (+ 610us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 04:07:28.094025 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:07:28.094065 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 04:07:28.094092 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:07:28.094254 (+ 162us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:07:28.094259 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:07:28.094262 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 04:07:28.094350 (+ 88us) client_negotiation.cc:240] Negotiation successful
0810 04:07:28.094460 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":107,"threads_started":1}
I20260810 04:07:28.095130 9837 heartbeater.cc:347] Connected to a master server at 127.31.32.62:33867
I20260810 04:07:28.095232 9837 heartbeater.cc:464] Registering TS with master...
I20260810 04:07:28.095422 9837 heartbeater.cc:511] Master 127.31.32.62:33867 requested a full tablet report, sending...
I20260810 04:07:28.095786 9485 ts_manager.cc:194] Registered new tserver with Master: db5931177a7e4e8288d676a379588c2c (127.31.32.2:34079)
I20260810 04:07:28.096295 9485 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.32.2:35029
W20260810 04:07:28.175297 9840 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:07:28.175480 9840 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:07:28.175500 9840 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:07:28.176666 9840 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:07:28.176705 9840 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:07:28.176729 9840 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.32.3
I20260810 04:07:28.177977 9840 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.32.20:34729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786334794676589-31872-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786334794676589-31872-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.32.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786334794676589-31872-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.32.3
--webserver_port=0
--tserver_master_addrs=127.31.32.62:33867
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9840
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.32.3
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786334794676589-31872-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:07:28.178200 9840 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:07:28.178479 9840 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:07:28.183141 9850 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:07:28.183208 9847 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:07:28.183231 9840 server_base.cc:1034] running on GCE node
W20260810 04:07:28.183192 9846 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:07:28.183593 9840 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:07:28.183849 9840 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:07:28.184995 9840 hybrid_clock.cc:648] HybridClock initialized: now 1786334848184983 us; error 28 us; skew 500 ppm
I20260810 04:07:28.186264 9840 webserver.cc:460] Webserver started at http://127.31.32.3:33983/ using document root <none> and password file <none>
I20260810 04:07:28.186501 9840 fs_manager.cc:359] Metadata directory not provided
I20260810 04:07:28.186573 9840 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:07:28.186725 9840 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:07:28.187778 9840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786334794676589-31872-0/minicluster-data/ts-2/data/instance:
uuid: "838e9a3a484b402494c1d2007cdb3224"
format_stamp: "Formatted at 2026-08-10 04:07:28 on dist-test-slave-8w9v"
server_key: "7b74908c14686cedb5edb6faa1e84f95"
server_key_iv: "6a95c3512be2cabbe2f0478fba8f61f5"
server_key_version: "encryptionkey@0"
I20260810 04:07:28.188200 9840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786334794676589-31872-0/minicluster-data/ts-2/wal/instance:
uuid: "838e9a3a484b402494c1d2007cdb3224"
format_stamp: "Formatted at 2026-08-10 04:07:28 on dist-test-slave-8w9v"
server_key: "7b74908c14686cedb5edb6faa1e84f95"
server_key_iv: "6a95c3512be2cabbe2f0478fba8f61f5"
server_key_version: "encryptionkey@0"
I20260810 04:07:28.189429 9840 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260810 04:07:28.190033 9856 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:07:28.190196 9840 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 04:07:28.190280 9840 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786334794676589-31872-0/minicluster-data/ts-2/data,/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786334794676589-31872-0/minicluster-data/ts-2/wal
uuid: "838e9a3a484b402494c1d2007cdb3224"
format_stamp: "Formatted at 2026-08-10 04:07:28 on dist-test-slave-8w9v"
server_key: "7b74908c14686cedb5edb6faa1e84f95"
server_key_iv: "6a95c3512be2cabbe2f0478fba8f61f5"
server_key_version: "encryptionkey@0"
I20260810 04:07:28.190346 9840 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786334794676589-31872-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786334794676589-31872-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786334794676589-31872-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:07:28.208171 9840 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:07:28.208490 9840 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:07:28.208638 9840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:07:28.208874 9840 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:07:28.209218 9840 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:07:28.209252 9840 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:07:28.209275 9840 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:07:28.209327 9840 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:07:28.214999 9840 rpc_server.cc:307] RPC server started. Bound to: 127.31.32.3:39543
I20260810 04:07:28.215054 9983 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.32.3:39543 every 8 connection(s)
I20260810 04:07:28.215356 9840 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786334794676589-31872-0/minicluster-data/ts-2/data/info.pb
I20260810 04:07:28.220407 31872 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskws89Io/build/release/bin/kudu as pid 9840
I20260810 04:07:28.220459 9524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:28.216084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.3:42911 (local address 127.31.32.62:33867)
0810 04:07:28.216182 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0810 04:07:28.216184 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:07:28.216396 (+ 212us) server_negotiation.cc:407] Connection header received
0810 04:07:28.216926 (+ 530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:07:28.216928 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:07:28.216939 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:07:28.216992 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:07:28.217650 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:28.218112 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:28.219015 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:28.219219 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:28.219291 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:28.220091 (+ 800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:07:28.220092 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:07:28.220093 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:07:28.220097 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:07:28.220150 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:07:28.220155 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:07:28.220242 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0810 04:07:28.220357 (+ 115us) server_negotiation.cc:299] Negotiation successful
0810 04:07:28.220382 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260810 04:07:28.220517 31872 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786334794676589-31872-0/minicluster-data/ts-2/wal/instance
I20260810 04:07:28.220665 9987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:28.216067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:33867 (local address 127.31.32.3:42911)
0810 04:07:28.216276 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0810 04:07:28.216307 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0810 04:07:28.216810 (+ 503us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:07:28.217059 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:07:28.217063 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:07:28.217072 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:07:28.217553 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:28.217557 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:28.218203 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:28.218204 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:28.218920 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:28.218921 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:28.219300 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:28.219301 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:28.219435 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:28.219951 (+ 516us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 04:07:28.219952 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:07:28.219980 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 04:07:28.220001 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:07:28.220247 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:07:28.220253 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:07:28.220256 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 04:07:28.220339 (+ 83us) client_negotiation.cc:240] Negotiation successful
0810 04:07:28.220450 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":101,"threads_started":1}
I20260810 04:07:28.220741 31872 external_mini_cluster.cc:1428] Setting key 515ebaa63e4246c79fc79cd08bc265bf
I20260810 04:07:28.221200 9984 heartbeater.cc:347] Connected to a master server at 127.31.32.62:33867
I20260810 04:07:28.221289 9984 heartbeater.cc:464] Registering TS with master...
I20260810 04:07:28.221487 9984 heartbeater.cc:511] Master 127.31.32.62:33867 requested a full tablet report, sending...
I20260810 04:07:28.221863 9485 ts_manager.cc:194] Registered new tserver with Master: 838e9a3a484b402494c1d2007cdb3224 (127.31.32.3:39543)
I20260810 04:07:28.222211 9485 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.32.3:42911
I20260810 04:07:28.232681 31872 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260810 04:07:28.234493 9524 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:07:28.234058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52768 (local address 127.31.32.62:33867)
0810 04:07:28.234110 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0810 04:07:28.234112 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:07:28.234211 (+ 99us) server_negotiation.cc:407] Connection header received
0810 04:07:28.234249 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:07:28.234249 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:07:28.234257 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:07:28.234315 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:07:28.234419 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52768: BlockingRecv error: recv got EOF from 127.0.0.1:52768 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
I20260810 04:07:28.234582 31872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskws89Io/build/release/bin/kudu with pid 9546
I20260810 04:07:28.241365 31872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskws89Io/build/release/bin/kudu with pid 9691
I20260810 04:07:28.248333 31872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskws89Io/build/release/bin/kudu with pid 9840
I20260810 04:07:28.254568 31872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskws89Io/build/release/bin/kudu with pid 9450
2026-08-10T04:07:28Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (625 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-08-10T04:07:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T04:07:28Z Disabled control of system clock
I20260810 04:07:28.282884 31872 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskws89Io/build/release/bin/kudu
/tmp/dist-test-taskws89Io/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786334794676589-31872-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786334794676589-31872-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786334794676589-31872-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786334794676589-31872-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.32.62:41199
--webserver_interface=127.31.32.62
--webserver_port=0
--builtin_ntp_servers=127.31.32.20:43735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.32.62:41199
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260810 04:07:28.363278 9998 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:07:28.363466 9998 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:07:28.363485 9998 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:07:28.364586 9998 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:07:28.364624 9998 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:07:28.364636 9998 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:07:28.364648 9998 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:07:28.364660 9998 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:07:28.365909 9998 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.32.20:43735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786334794676589-31872-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786334794676589-31872-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.32.62:41199
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.32.62:41199
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786334794676589-31872-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.32.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9998
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786334794676589-31872-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:07:28.366089 9998 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:07:28.366310 9998 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:07:28.371155 10004 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:07:28.371174 10006 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:07:28.371158 9998 server_base.cc:1034] running on GCE node
W20260810 04:07:28.371214 10003 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:07:28.371618 9998 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:07:28.371919 9998 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:07:28.373085 9998 hybrid_clock.cc:648] HybridClock initialized: now 1786334848373062 us; error 54 us; skew 500 ppm
I20260810 04:07:28.374468 9998 webserver.cc:460] Webserver started at http://127.31.32.62:45533/ using document root <none> and password file <none>
I20260810 04:07:28.374734 9998 fs_manager.cc:359] Metadata directory not provided
I20260810 04:07:28.374805 9998 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:07:28.374950 9998 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:07:28.376067 9998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786334794676589-31872-0/minicluster-data/master-0/data/instance:
uuid: "90d9f3415c314fbaa793cf7cce11fbb5"
format_stamp: "Formatted at 2026-08-10 04:07:28 on dist-test-slave-8w9v"
server_key: "d4412288f7376ebdb84521aecdfe0156"
server_key_iv: "e575e657d017768027d15bf218727750"
server_key_version: "encryptionkey@0"
I20260810 04:07:28.376500 9998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786334794676589-31872-0/minicluster-data/master-0/wal/instance:
uuid: "90d9f3415c314fbaa793cf7cce11fbb5"
format_stamp: "Formatted at 2026-08-10 04:07:28 on dist-test-slave-8w9v"
server_key: "d4412288f7376ebdb84521aecdfe0156"
server_key_iv: "e575e657d017768027d15bf218727750"
server_key_version: "encryptionkey@0"
I20260810 04:07:28.377883 9998 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:07:28.378647 10012 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:07:28.378820 9998 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:07:28.378925 9998 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786334794676589-31872-0/minicluster-data/master-0/data,/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786334794676589-31872-0/minicluster-data/master-0/wal
uuid: "90d9f3415c314fbaa793cf7cce11fbb5"
format_stamp: "Formatted at 2026-08-10 04:07:28 on dist-test-slave-8w9v"
server_key: "d4412288f7376ebdb84521aecdfe0156"
server_key_iv: "e575e657d017768027d15bf218727750"
server_key_version: "encryptionkey@0"
I20260810 04:07:28.378997 9998 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786334794676589-31872-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786334794676589-31872-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786334794676589-31872-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:07:28.382311 9998 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:07:28.382575 9998 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:07:28.382714 9998 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:07:28.386595 9998 rpc_server.cc:307] RPC server started. Bound to: 127.31.32.62:41199
I20260810 04:07:28.386626 10071 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.32.62:41199 every 8 connection(s)
I20260810 04:07:28.386945 9998 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786334794676589-31872-0/minicluster-data/master-0/data/info.pb
I20260810 04:07:28.387650 10072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:07:28.390465 10072 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90d9f3415c314fbaa793cf7cce11fbb5: Bootstrap starting.
I20260810 04:07:28.391039 10072 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 90d9f3415c314fbaa793cf7cce11fbb5: Neither blocks nor log segments found. Creating new log.
I20260810 04:07:28.391285 10072 log.cc:824] T 00000000000000000000000000000000 P 90d9f3415c314fbaa793cf7cce11fbb5: Log is configured to *not* fsync() on all Append() calls
I20260810 04:07:28.391921 10072 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90d9f3415c314fbaa793cf7cce11fbb5: No bootstrap required, opened a new log
I20260810 04:07:28.393544 10072 raft_consensus.cc:348] T 00000000000000000000000000000000 P 90d9f3415c314fbaa793cf7cce11fbb5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90d9f3415c314fbaa793cf7cce11fbb5" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 41199 } }
I20260810 04:07:28.393676 10072 raft_consensus.cc:374] T 00000000000000000000000000000000 P 90d9f3415c314fbaa793cf7cce11fbb5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:07:28.393699 10072 raft_consensus.cc:729] T 00000000000000000000000000000000 P 90d9f3415c314fbaa793cf7cce11fbb5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90d9f3415c314fbaa793cf7cce11fbb5, State: Initialized, Role: FOLLOWER
I20260810 04:07:28.393798 10072 consensus_queue.cc:260] T 00000000000000000000000000000000 P 90d9f3415c314fbaa793cf7cce11fbb5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90d9f3415c314fbaa793cf7cce11fbb5" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 41199 } }
I20260810 04:07:28.393846 10072 raft_consensus.cc:388] T 00000000000000000000000000000000 P 90d9f3415c314fbaa793cf7cce11fbb5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 04:07:28.393905 10072 raft_consensus.cc:482] T 00000000000000000000000000000000 P 90d9f3415c314fbaa793cf7cce11fbb5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 04:07:28.393950 10072 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 90d9f3415c314fbaa793cf7cce11fbb5 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:07:28.394474 10072 raft_consensus.cc:504] T 00000000000000000000000000000000 P 90d9f3415c314fbaa793cf7cce11fbb5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90d9f3415c314fbaa793cf7cce11fbb5" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 41199 } }
I20260810 04:07:28.394559 10072 leader_election.cc:302] T 00000000000000000000000000000000 P 90d9f3415c314fbaa793cf7cce11fbb5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 90d9f3415c314fbaa793cf7cce11fbb5; no voters:
I20260810 04:07:28.394762 10072 leader_election.cc:288] T 00000000000000000000000000000000 P 90d9f3415c314fbaa793cf7cce11fbb5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 04:07:28.394855 10075 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 90d9f3415c314fbaa793cf7cce11fbb5 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:07:28.395041 10072 sys_catalog.cc:564] T 00000000000000000000000000000000 P 90d9f3415c314fbaa793cf7cce11fbb5 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:07:28.395118 10075 raft_consensus.cc:686] T 00000000000000000000000000000000 P 90d9f3415c314fbaa793cf7cce11fbb5 [term 1 LEADER]: Becoming Leader. State: Replica: 90d9f3415c314fbaa793cf7cce11fbb5, State: Running, Role: LEADER
I20260810 04:07:28.395222 10075 consensus_queue.cc:237] T 00000000000000000000000000000000 P 90d9f3415c314fbaa793cf7cce11fbb5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90d9f3415c314fbaa793cf7cce11fbb5" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 41199 } }
I20260810 04:07:28.395696 10076 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90d9f3415c314fbaa793cf7cce11fbb5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "90d9f3415c314fbaa793cf7cce11fbb5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90d9f3415c314fbaa793cf7cce11fbb5" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 41199 } } }
I20260810 04:07:28.395797 10076 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90d9f3415c314fbaa793cf7cce11fbb5 [sys.catalog]: This master's current role is: LEADER
I20260810 04:07:28.395931 10077 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90d9f3415c314fbaa793cf7cce11fbb5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 90d9f3415c314fbaa793cf7cce11fbb5. Latest consensus state: current_term: 1 leader_uuid: "90d9f3415c314fbaa793cf7cce11fbb5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90d9f3415c314fbaa793cf7cce11fbb5" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 41199 } } }
I20260810 04:07:28.396030 10077 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90d9f3415c314fbaa793cf7cce11fbb5 [sys.catalog]: This master's current role is: LEADER
I20260810 04:07:28.396469 10084 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 04:07:28.397004 10084 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 04:07:28.397109 31872 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskws89Io/build/release/bin/kudu as pid 9998
I20260810 04:07:28.397204 31872 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786334794676589-31872-0/minicluster-data/master-0/wal/instance
I20260810 04:07:28.397410 31872 external_mini_cluster.cc:1428] Setting key fe6b08a2dd1d4497926f0b84e7d42b7c
I20260810 04:07:28.398762 10084 catalog_manager.cc:1347] Generated new cluster ID: 35ffb11946ff4560a9afc50037bb1386
I20260810 04:07:28.398825 10084 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 04:07:28.400462 10096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:28.397903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34328 (local address 127.31.32.62:41199)
0810 04:07:28.398102 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0810 04:07:28.398113 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 04:07:28.398192 (+ 79us) server_negotiation.cc:407] Connection header received
0810 04:07:28.398418 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:07:28.398426 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:07:28.398473 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:07:28.399467 (+ 994us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:07:28.399831 (+ 364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:07:28.399834 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:07:28.399835 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:07:28.399838 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:07:28.399885 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:07:28.399897 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:07:28.400195 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0810 04:07:28.400225 (+ 30us) server_negotiation.cc:299] Negotiation successful
0810 04:07:28.400291 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":84,"threads_started":1}
I20260810 04:07:28.406576 10084 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 04:07:28.406749 10084 catalog_manager.cc:1504] Loading token signing keys...
I20260810 04:07:28.412320 10084 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 90d9f3415c314fbaa793cf7cce11fbb5: Generated new TSK 0
I20260810 04:07:28.412528 10084 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 04:07:28.413926 31872 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskws89Io/build/release/bin/kudu
/tmp/dist-test-taskws89Io/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786334794676589-31872-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786334794676589-31872-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786334794676589-31872-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786334794676589-31872-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.32.1:0
--local_ip_for_outbound_sockets=127.31.32.1
--webserver_interface=127.31.32.1
--webserver_port=0
--tserver_master_addrs=127.31.32.62:41199
--builtin_ntp_servers=127.31.32.20:43735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260810 04:07:28.495858 10098 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:07:28.496073 10098 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:07:28.496095 10098 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:07:28.497361 10098 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:07:28.497419 10098 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:07:28.497445 10098 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.32.1
I20260810 04:07:28.498785 10098 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.32.20:43735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786334794676589-31872-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786334794676589-31872-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.32.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786334794676589-31872-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.32.1
--webserver_port=0
--tserver_master_addrs=127.31.32.62:41199
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10098
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.32.1
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786334794676589-31872-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:07:28.499023 10098 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:07:28.499334 10098 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:07:28.504596 10103 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:07:28.504700 10098 server_base.cc:1034] running on GCE node
W20260810 04:07:28.504596 10106 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:07:28.504908 10104 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:07:28.505132 10098 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:07:28.505421 10098 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:07:28.506572 10098 hybrid_clock.cc:648] HybridClock initialized: now 1786334848506552 us; error 38 us; skew 500 ppm
I20260810 04:07:28.507896 10098 webserver.cc:460] Webserver started at http://127.31.32.1:45437/ using document root <none> and password file <none>
I20260810 04:07:28.508201 10098 fs_manager.cc:359] Metadata directory not provided
I20260810 04:07:28.508257 10098 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:07:28.508428 10098 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:07:28.509513 10098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786334794676589-31872-0/minicluster-data/ts-0/data/instance:
uuid: "028f7c3db05c4d8c8cec4be49b2c8678"
format_stamp: "Formatted at 2026-08-10 04:07:28 on dist-test-slave-8w9v"
server_key: "5cf25163ad853e26745703c748c794c4"
server_key_iv: "65253697dde3e28a0ee8bb4f007cfce5"
server_key_version: "encryptionkey@0"
I20260810 04:07:28.509859 10098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786334794676589-31872-0/minicluster-data/ts-0/wal/instance:
uuid: "028f7c3db05c4d8c8cec4be49b2c8678"
format_stamp: "Formatted at 2026-08-10 04:07:28 on dist-test-slave-8w9v"
server_key: "5cf25163ad853e26745703c748c794c4"
server_key_iv: "65253697dde3e28a0ee8bb4f007cfce5"
server_key_version: "encryptionkey@0"
I20260810 04:07:28.511161 10098 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 04:07:28.511916 10115 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:07:28.512066 10098 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 04:07:28.512189 10098 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786334794676589-31872-0/minicluster-data/ts-0/data,/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786334794676589-31872-0/minicluster-data/ts-0/wal
uuid: "028f7c3db05c4d8c8cec4be49b2c8678"
format_stamp: "Formatted at 2026-08-10 04:07:28 on dist-test-slave-8w9v"
server_key: "5cf25163ad853e26745703c748c794c4"
server_key_iv: "65253697dde3e28a0ee8bb4f007cfce5"
server_key_version: "encryptionkey@0"
I20260810 04:07:28.512264 10098 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786334794676589-31872-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786334794676589-31872-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786334794676589-31872-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:07:28.515499 10098 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:07:28.515780 10098 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:07:28.515914 10098 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:07:28.516202 10098 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:07:28.516618 10098 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:07:28.516655 10098 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:07:28.516676 10098 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:07:28.516688 10098 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:07:28.523821 10098 rpc_server.cc:307] RPC server started. Bound to: 127.31.32.1:41567
I20260810 04:07:28.523910 10244 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.32.1:41567 every 8 connection(s)
I20260810 04:07:28.524303 10098 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786334794676589-31872-0/minicluster-data/ts-0/data/info.pb
I20260810 04:07:28.527354 10096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:28.525075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.1:38643 (local address 127.31.32.62:41199)
0810 04:07:28.525117 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0810 04:07:28.525119 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:07:28.525345 (+ 226us) server_negotiation.cc:407] Connection header received
0810 04:07:28.525854 (+ 509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:07:28.525855 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:07:28.525863 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:07:28.525908 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:07:28.527095 (+ 1187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:07:28.527096 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:07:28.527096 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:07:28.527102 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:07:28.527119 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:07:28.527123 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:07:28.527182 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0810 04:07:28.527271 (+ 89us) server_negotiation.cc:299] Negotiation successful
0810 04:07:28.527288 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260810 04:07:28.527577 10248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:28.525042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:41199 (local address 127.31.32.1:38643)
0810 04:07:28.525226 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0810 04:07:28.525258 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0810 04:07:28.525734 (+ 476us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:07:28.525993 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:07:28.525996 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:07:28.526004 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:07:28.526942 (+ 938us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 04:07:28.526942 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:07:28.526970 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 04:07:28.526997 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:07:28.527182 (+ 185us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:07:28.527188 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:07:28.527191 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 04:07:28.527281 (+ 90us) client_negotiation.cc:240] Negotiation successful
0810 04:07:28.527349 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":81,"threads_started":1}
I20260810 04:07:28.528106 10245 heartbeater.cc:347] Connected to a master server at 127.31.32.62:41199
I20260810 04:07:28.528257 10245 heartbeater.cc:464] Registering TS with master...
I20260810 04:07:28.528370 10245 heartbeater.cc:511] Master 127.31.32.62:41199 requested a full tablet report, sending...
I20260810 04:07:28.528851 10029 ts_manager.cc:194] Registered new tserver with Master: 028f7c3db05c4d8c8cec4be49b2c8678 (127.31.32.1:41567)
I20260810 04:07:28.529430 31872 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskws89Io/build/release/bin/kudu as pid 10098
I20260810 04:07:28.529534 31872 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786334794676589-31872-0/minicluster-data/ts-0/wal/instance
I20260810 04:07:28.529729 31872 external_mini_cluster.cc:1428] Setting key 76d87b4987af140c5e7d29ed62edbeee
I20260810 04:07:28.530508 31872 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskws89Io/build/release/bin/kudu
/tmp/dist-test-taskws89Io/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786334794676589-31872-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786334794676589-31872-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786334794676589-31872-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786334794676589-31872-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.32.2:0
--local_ip_for_outbound_sockets=127.31.32.2
--webserver_interface=127.31.32.2
--webserver_port=0
--tserver_master_addrs=127.31.32.62:41199
--builtin_ntp_servers=127.31.32.20:43735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260810 04:07:28.612375 10250 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:07:28.612632 10250 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:07:28.612668 10250 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:07:28.613868 10250 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:07:28.613936 10250 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:07:28.613979 10250 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.32.2
I20260810 04:07:28.615365 10250 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.32.20:43735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786334794676589-31872-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786334794676589-31872-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.32.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786334794676589-31872-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.32.2
--webserver_port=0
--tserver_master_addrs=127.31.32.62:41199
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10250
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.32.2
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786334794676589-31872-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:07:28.615633 10250 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:07:28.615954 10250 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:07:28.620971 10257 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:07:28.620987 10256 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:07:28.621012 10259 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:07:28.621210 10250 server_base.cc:1034] running on GCE node
I20260810 04:07:28.621481 10250 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:07:28.621714 10250 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:07:28.622887 10250 hybrid_clock.cc:648] HybridClock initialized: now 1786334848622879 us; error 31 us; skew 500 ppm
I20260810 04:07:28.624166 10250 webserver.cc:460] Webserver started at http://127.31.32.2:34945/ using document root <none> and password file <none>
I20260810 04:07:28.624415 10250 fs_manager.cc:359] Metadata directory not provided
I20260810 04:07:28.624472 10250 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:07:28.624595 10250 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:07:28.625725 10250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786334794676589-31872-0/minicluster-data/ts-1/data/instance:
uuid: "be067254b3f14960a80f6680188c2bbe"
format_stamp: "Formatted at 2026-08-10 04:07:28 on dist-test-slave-8w9v"
server_key: "ed8b20b2394a04ce86024f4deed826e8"
server_key_iv: "12731db820f2113dbb225dbe17dd9d2e"
server_key_version: "encryptionkey@0"
I20260810 04:07:28.626032 10250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786334794676589-31872-0/minicluster-data/ts-1/wal/instance:
uuid: "be067254b3f14960a80f6680188c2bbe"
format_stamp: "Formatted at 2026-08-10 04:07:28 on dist-test-slave-8w9v"
server_key: "ed8b20b2394a04ce86024f4deed826e8"
server_key_iv: "12731db820f2113dbb225dbe17dd9d2e"
server_key_version: "encryptionkey@0"
I20260810 04:07:28.627274 10250 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 04:07:28.627955 10265 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:07:28.628188 10250 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:07:28.628280 10250 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786334794676589-31872-0/minicluster-data/ts-1/data,/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786334794676589-31872-0/minicluster-data/ts-1/wal
uuid: "be067254b3f14960a80f6680188c2bbe"
format_stamp: "Formatted at 2026-08-10 04:07:28 on dist-test-slave-8w9v"
server_key: "ed8b20b2394a04ce86024f4deed826e8"
server_key_iv: "12731db820f2113dbb225dbe17dd9d2e"
server_key_version: "encryptionkey@0"
I20260810 04:07:28.628334 10250 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786334794676589-31872-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786334794676589-31872-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786334794676589-31872-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:07:28.631538 10250 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:07:28.631754 10250 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:07:28.631863 10250 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:07:28.632017 10250 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:07:28.632352 10250 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:07:28.632413 10250 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:07:28.632454 10250 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:07:28.632481 10250 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:07:28.638444 10250 rpc_server.cc:307] RPC server started. Bound to: 127.31.32.2:38701
I20260810 04:07:28.638489 10391 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.32.2:38701 every 8 connection(s)
I20260810 04:07:28.638792 10250 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786334794676589-31872-0/minicluster-data/ts-1/data/info.pb
I20260810 04:07:28.642005 10096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:28.639474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.2:33397 (local address 127.31.32.62:41199)
0810 04:07:28.639545 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0810 04:07:28.639547 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:07:28.639828 (+ 281us) server_negotiation.cc:407] Connection header received
0810 04:07:28.640418 (+ 590us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:07:28.640419 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:07:28.640427 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:07:28.640491 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:07:28.641721 (+ 1230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:07:28.641723 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:07:28.641724 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:07:28.641728 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:07:28.641747 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:07:28.641751 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:07:28.641806 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0810 04:07:28.641908 (+ 102us) server_negotiation.cc:299] Negotiation successful
0810 04:07:28.641933 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260810 04:07:28.642196 10394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:28.639473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:41199 (local address 127.31.32.2:33397)
0810 04:07:28.639682 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0810 04:07:28.639721 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0810 04:07:28.640309 (+ 588us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:07:28.640560 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:07:28.640564 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:07:28.640573 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:07:28.641558 (+ 985us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 04:07:28.641559 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:07:28.641587 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 04:07:28.641615 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:07:28.641815 (+ 200us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:07:28.641821 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:07:28.641824 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 04:07:28.641919 (+ 95us) client_negotiation.cc:240] Negotiation successful
0810 04:07:28.641982 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":120,"threads_started":1}
I20260810 04:07:28.642653 10392 heartbeater.cc:347] Connected to a master server at 127.31.32.62:41199
I20260810 04:07:28.642728 10392 heartbeater.cc:464] Registering TS with master...
I20260810 04:07:28.642921 10392 heartbeater.cc:511] Master 127.31.32.62:41199 requested a full tablet report, sending...
I20260810 04:07:28.643270 10029 ts_manager.cc:194] Registered new tserver with Master: be067254b3f14960a80f6680188c2bbe (127.31.32.2:38701)
I20260810 04:07:28.645654 31872 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskws89Io/build/release/bin/kudu as pid 10250
I20260810 04:07:28.645768 31872 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786334794676589-31872-0/minicluster-data/ts-1/wal/instance
I20260810 04:07:28.645969 31872 external_mini_cluster.cc:1428] Setting key c7a10a9813602ee4ac286567c4f20cc2
I20260810 04:07:28.646867 31872 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskws89Io/build/release/bin/kudu
/tmp/dist-test-taskws89Io/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786334794676589-31872-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786334794676589-31872-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786334794676589-31872-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786334794676589-31872-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.32.3:0
--local_ip_for_outbound_sockets=127.31.32.3
--webserver_interface=127.31.32.3
--webserver_port=0
--tserver_master_addrs=127.31.32.62:41199
--builtin_ntp_servers=127.31.32.20:43735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260810 04:07:28.730227 10395 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:07:28.730412 10395 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:07:28.730433 10395 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:07:28.731600 10395 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:07:28.731655 10395 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:07:28.731698 10395 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.32.3
I20260810 04:07:28.733134 10395 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.32.20:43735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786334794676589-31872-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786334794676589-31872-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.32.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786334794676589-31872-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.32.3
--webserver_port=0
--tserver_master_addrs=127.31.32.62:41199
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10395
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.32.3
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786334794676589-31872-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:07:28.733372 10395 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:07:28.733657 10395 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:07:28.738603 10404 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:07:28.738646 10401 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:07:28.738692 10395 server_base.cc:1034] running on GCE node
W20260810 04:07:28.738600 10402 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:07:28.739001 10395 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:07:28.739246 10395 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:07:28.740409 10395 hybrid_clock.cc:648] HybridClock initialized: now 1786334848740389 us; error 37 us; skew 500 ppm
I20260810 04:07:28.741735 10395 webserver.cc:460] Webserver started at http://127.31.32.3:43009/ using document root <none> and password file <none>
I20260810 04:07:28.741979 10395 fs_manager.cc:359] Metadata directory not provided
I20260810 04:07:28.742054 10395 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:07:28.742213 10395 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:07:28.743216 10395 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786334794676589-31872-0/minicluster-data/ts-2/data/instance:
uuid: "cda5610e41444d43823131c6853e8ea1"
format_stamp: "Formatted at 2026-08-10 04:07:28 on dist-test-slave-8w9v"
server_key: "08bc2cd2666a39c317c73c2606076f89"
server_key_iv: "2e4f9824181b39ccd8879c6650e4a7b8"
server_key_version: "encryptionkey@0"
I20260810 04:07:28.743592 10395 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786334794676589-31872-0/minicluster-data/ts-2/wal/instance:
uuid: "cda5610e41444d43823131c6853e8ea1"
format_stamp: "Formatted at 2026-08-10 04:07:28 on dist-test-slave-8w9v"
server_key: "08bc2cd2666a39c317c73c2606076f89"
server_key_iv: "2e4f9824181b39ccd8879c6650e4a7b8"
server_key_version: "encryptionkey@0"
I20260810 04:07:28.744989 10395 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 04:07:28.745718 10411 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:07:28.745888 10395 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 04:07:28.745990 10395 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786334794676589-31872-0/minicluster-data/ts-2/data,/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786334794676589-31872-0/minicluster-data/ts-2/wal
uuid: "cda5610e41444d43823131c6853e8ea1"
format_stamp: "Formatted at 2026-08-10 04:07:28 on dist-test-slave-8w9v"
server_key: "08bc2cd2666a39c317c73c2606076f89"
server_key_iv: "2e4f9824181b39ccd8879c6650e4a7b8"
server_key_version: "encryptionkey@0"
I20260810 04:07:28.746070 10395 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786334794676589-31872-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786334794676589-31872-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786334794676589-31872-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:07:28.749353 10395 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:07:28.749594 10395 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:07:28.749734 10395 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:07:28.749933 10395 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:07:28.750218 10395 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:07:28.750272 10395 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:07:28.750309 10395 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:07:28.750342 10395 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:07:28.755900 10395 rpc_server.cc:307] RPC server started. Bound to: 127.31.32.3:45519
I20260810 04:07:28.755937 10537 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.32.3:45519 every 8 connection(s)
I20260810 04:07:28.756322 10395 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786334794676589-31872-0/minicluster-data/ts-2/data/info.pb
I20260810 04:07:28.759109 10096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:28.757084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.3:43229 (local address 127.31.32.62:41199)
0810 04:07:28.757128 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0810 04:07:28.757130 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:07:28.757342 (+ 212us) server_negotiation.cc:407] Connection header received
0810 04:07:28.757741 (+ 399us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:07:28.757742 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:07:28.757747 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:07:28.757803 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:07:28.758928 (+ 1125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:07:28.758929 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:07:28.758929 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:07:28.758932 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:07:28.758942 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:07:28.758945 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:07:28.758988 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0810 04:07:28.759029 (+ 41us) server_negotiation.cc:299] Negotiation successful
0810 04:07:28.759040 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260810 04:07:28.759320 10540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:28.757056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:41199 (local address 127.31.32.3:43229)
0810 04:07:28.757221 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0810 04:07:28.757257 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0810 04:07:28.757663 (+ 406us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:07:28.757850 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:07:28.757853 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:07:28.757861 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:07:28.758805 (+ 944us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 04:07:28.758806 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:07:28.758833 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 04:07:28.758858 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:07:28.758985 (+ 127us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:07:28.758987 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:07:28.758989 (+ 2us) client_negotiation.cc:769] Sending connection context
0810 04:07:28.759035 (+ 46us) client_negotiation.cc:240] Negotiation successful
0810 04:07:28.759095 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":78,"threads_started":1}
I20260810 04:07:28.759806 10538 heartbeater.cc:347] Connected to a master server at 127.31.32.62:41199
I20260810 04:07:28.759892 10538 heartbeater.cc:464] Registering TS with master...
I20260810 04:07:28.759995 10538 heartbeater.cc:511] Master 127.31.32.62:41199 requested a full tablet report, sending...
I20260810 04:07:28.760366 10029 ts_manager.cc:194] Registered new tserver with Master: cda5610e41444d43823131c6853e8ea1 (127.31.32.3:45519)
I20260810 04:07:28.762152 31872 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskws89Io/build/release/bin/kudu as pid 10395
I20260810 04:07:28.762228 31872 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786334794676589-31872-0/minicluster-data/ts-2/wal/instance
I20260810 04:07:28.762434 31872 external_mini_cluster.cc:1428] Setting key 229606f84c4013e93ded160c2c2d45a3
I20260810 04:07:28.763288 31872 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260810 04:07:28.765132 10096 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:07:28.764604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34344 (local address 127.31.32.62:41199)
0810 04:07:28.764639 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0810 04:07:28.764641 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:07:28.764797 (+ 156us) server_negotiation.cc:407] Connection header received
0810 04:07:28.764873 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:07:28.764874 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:07:28.764879 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:07:28.764944 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:07:28.765070 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34344: BlockingRecv error: recv got EOF from 127.0.0.1:34344 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
I20260810 04:07:28.765211 31872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskws89Io/build/release/bin/kudu with pid 10098
I20260810 04:07:28.779572 31872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskws89Io/build/release/bin/kudu with pid 10250
I20260810 04:07:28.786616 31872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskws89Io/build/release/bin/kudu with pid 10395
I20260810 04:07:28.792570 31872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskws89Io/build/release/bin/kudu with pid 9998
2026-08-10T04:07:28Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (538 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786334794676589-31872-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 04:07:28 dist-test-slave-8w9v krb5kdc[10552](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 04:07:28 dist-test-slave-8w9v krb5kdc[10552](info): set up 2 sockets
Aug 10 04:07:28 dist-test-slave-8w9v krb5kdc[10552](info): commencing operation
krb5kdc: starting...
W20260810 04:07:29.865407 31872 mini_kdc.cc:121] Time spent starting KDC: real 1.052s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786334794676589-31872-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786334794676589-31872-0/krb5kdc/test-user.keytab.
Aug 10 04:07:29 dist-test-slave-8w9v krb5kdc[10552](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334849, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T04:07:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T04:07:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.32.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.32.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.32.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.62.keytab.
Entry for principal kudu/127.31.32.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.62.keytab.
I20260810 04:07:29.980830 31872 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskws89Io/build/release/bin/kudu
/tmp/dist-test-taskws89Io/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786334794676589-31872-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786334794676589-31872-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786334794676589-31872-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786334794676589-31872-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.32.62:44097
--webserver_interface=127.31.32.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.62.keytab
--principal=kudu/127.31.32.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.32.20:38741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.32.62:44097
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786334794676589-31872-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786334794676589-31872-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786334794676589-31872-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:07:30.061523 10567 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:07:30.061733 10567 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:07:30.061754 10567 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:07:30.062892 10567 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:07:30.062934 10567 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:07:30.062947 10567 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:07:30.062960 10567 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:07:30.062997 10567 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:07:30.064438 10567 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.32.20:38741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786334794676589-31872-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786334794676589-31872-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.32.62:44097
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.62.keytab
--principal=kudu/127.31.32.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.32.62:44097
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786334794676589-31872-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.32.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10567
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786334794676589-31872-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:07:30.064682 10567 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:07:30.064911 10567 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:07:30.069949 10574 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:07:30.070003 10576 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:07:30.069945 10573 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:07:30.070118 10567 server_base.cc:1034] running on GCE node
I20260810 04:07:30.070472 10567 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:07:30.070817 10567 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:07:30.071985 10567 hybrid_clock.cc:648] HybridClock initialized: now 1786334850071943 us; error 62 us; skew 500 ppm
Aug 10 04:07:30 dist-test-slave-8w9v krb5kdc[10552](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334850, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:07:30.074687 10567 init.cc:377] Logged in from keytab as kudu/127.31.32.62@KRBTEST.COM (short username kudu)
I20260810 04:07:30.075198 10567 webserver.cc:460] Webserver started at http://127.31.32.62:36801/ using document root <none> and password file <none>
I20260810 04:07:30.075462 10567 fs_manager.cc:359] Metadata directory not provided
I20260810 04:07:30.075536 10567 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:07:30.075695 10567 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:07:30.076900 10567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786334794676589-31872-0/minicluster-data/master-0/data/instance:
uuid: "416c03f2ba5545a3b640aec05f1027ac"
format_stamp: "Formatted at 2026-08-10 04:07:30 on dist-test-slave-8w9v"
server_key: "0d5b0a87c11707172f2e78ab5fa08caa"
server_key_iv: "3136427a8e21db515d62762d68972fed"
server_key_version: "encryptionkey@0"
I20260810 04:07:30.077309 10567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786334794676589-31872-0/minicluster-data/master-0/wal/instance:
uuid: "416c03f2ba5545a3b640aec05f1027ac"
format_stamp: "Formatted at 2026-08-10 04:07:30 on dist-test-slave-8w9v"
server_key: "0d5b0a87c11707172f2e78ab5fa08caa"
server_key_iv: "3136427a8e21db515d62762d68972fed"
server_key_version: "encryptionkey@0"
I20260810 04:07:30.078771 10567 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 04:07:30.079547 10583 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:07:30.079738 10567 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:07:30.079838 10567 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786334794676589-31872-0/minicluster-data/master-0/data,/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786334794676589-31872-0/minicluster-data/master-0/wal
uuid: "416c03f2ba5545a3b640aec05f1027ac"
format_stamp: "Formatted at 2026-08-10 04:07:30 on dist-test-slave-8w9v"
server_key: "0d5b0a87c11707172f2e78ab5fa08caa"
server_key_iv: "3136427a8e21db515d62762d68972fed"
server_key_version: "encryptionkey@0"
I20260810 04:07:30.079924 10567 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786334794676589-31872-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786334794676589-31872-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786334794676589-31872-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:07:30.097644 10567 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:07:30.099545 10567 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:07:30.099713 10567 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:07:30.104578 10567 rpc_server.cc:307] RPC server started. Bound to: 127.31.32.62:44097
I20260810 04:07:30.104628 10644 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.32.62:44097 every 8 connection(s)
I20260810 04:07:30.104998 10567 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786334794676589-31872-0/minicluster-data/master-0/data/info.pb
I20260810 04:07:30.105477 31872 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskws89Io/build/release/bin/kudu as pid 10567
I20260810 04:07:30.105614 31872 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786334794676589-31872-0/minicluster-data/master-0/wal/instance
I20260810 04:07:30.105825 31872 external_mini_cluster.cc:1428] Setting key 277120adeb3d2d3d05045281758aa680
I20260810 04:07:30.106287 10645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:07:30.109562 10645 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 416c03f2ba5545a3b640aec05f1027ac: Bootstrap starting.
I20260810 04:07:30.110188 10645 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 416c03f2ba5545a3b640aec05f1027ac: Neither blocks nor log segments found. Creating new log.
I20260810 04:07:30.110482 10645 log.cc:824] T 00000000000000000000000000000000 P 416c03f2ba5545a3b640aec05f1027ac: Log is configured to *not* fsync() on all Append() calls
I20260810 04:07:30.111184 10645 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 416c03f2ba5545a3b640aec05f1027ac: No bootstrap required, opened a new log
Aug 10 04:07:30 dist-test-slave-8w9v krb5kdc[10552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334849, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.32.62@KRBTEST.COM
I20260810 04:07:30.113210 10645 raft_consensus.cc:348] T 00000000000000000000000000000000 P 416c03f2ba5545a3b640aec05f1027ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "416c03f2ba5545a3b640aec05f1027ac" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 44097 } }
I20260810 04:07:30.113379 10645 raft_consensus.cc:374] T 00000000000000000000000000000000 P 416c03f2ba5545a3b640aec05f1027ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:07:30.113446 10645 raft_consensus.cc:729] T 00000000000000000000000000000000 P 416c03f2ba5545a3b640aec05f1027ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 416c03f2ba5545a3b640aec05f1027ac, State: Initialized, Role: FOLLOWER
I20260810 04:07:30.113613 10645 consensus_queue.cc:260] T 00000000000000000000000000000000 P 416c03f2ba5545a3b640aec05f1027ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "416c03f2ba5545a3b640aec05f1027ac" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 44097 } }
I20260810 04:07:30.113732 10645 raft_consensus.cc:388] T 00000000000000000000000000000000 P 416c03f2ba5545a3b640aec05f1027ac [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 04:07:30.113782 10645 raft_consensus.cc:482] T 00000000000000000000000000000000 P 416c03f2ba5545a3b640aec05f1027ac [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 04:07:30.113874 10645 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 416c03f2ba5545a3b640aec05f1027ac [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:07:30.114655 10645 raft_consensus.cc:504] T 00000000000000000000000000000000 P 416c03f2ba5545a3b640aec05f1027ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "416c03f2ba5545a3b640aec05f1027ac" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 44097 } }
I20260810 04:07:30.114777 10645 leader_election.cc:302] T 00000000000000000000000000000000 P 416c03f2ba5545a3b640aec05f1027ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 416c03f2ba5545a3b640aec05f1027ac; no voters:
I20260810 04:07:30.114948 10645 leader_election.cc:288] T 00000000000000000000000000000000 P 416c03f2ba5545a3b640aec05f1027ac [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 04:07:30.115015 10650 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 416c03f2ba5545a3b640aec05f1027ac [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:07:30.115176 10650 raft_consensus.cc:686] T 00000000000000000000000000000000 P 416c03f2ba5545a3b640aec05f1027ac [term 1 LEADER]: Becoming Leader. State: Replica: 416c03f2ba5545a3b640aec05f1027ac, State: Running, Role: LEADER
I20260810 04:07:30.115212 10645 sys_catalog.cc:564] T 00000000000000000000000000000000 P 416c03f2ba5545a3b640aec05f1027ac [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:07:30.115317 10650 consensus_queue.cc:237] T 00000000000000000000000000000000 P 416c03f2ba5545a3b640aec05f1027ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "416c03f2ba5545a3b640aec05f1027ac" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 44097 } }
I20260810 04:07:30.115717 10653 sys_catalog.cc:455] T 00000000000000000000000000000000 P 416c03f2ba5545a3b640aec05f1027ac [sys.catalog]: SysCatalogTable state changed. Reason: New leader 416c03f2ba5545a3b640aec05f1027ac. Latest consensus state: current_term: 1 leader_uuid: "416c03f2ba5545a3b640aec05f1027ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "416c03f2ba5545a3b640aec05f1027ac" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 44097 } } }
I20260810 04:07:30.115805 10653 sys_catalog.cc:458] T 00000000000000000000000000000000 P 416c03f2ba5545a3b640aec05f1027ac [sys.catalog]: This master's current role is: LEADER
I20260810 04:07:30.115723 10652 sys_catalog.cc:455] T 00000000000000000000000000000000 P 416c03f2ba5545a3b640aec05f1027ac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "416c03f2ba5545a3b640aec05f1027ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "416c03f2ba5545a3b640aec05f1027ac" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 44097 } } }
I20260810 04:07:30.116014 10652 sys_catalog.cc:458] T 00000000000000000000000000000000 P 416c03f2ba5545a3b640aec05f1027ac [sys.catalog]: This master's current role is: LEADER
I20260810 04:07:30.116014 10658 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 04:07:30.116683 10658 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 04:07:30.117738 10648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:30.106390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39558 (local address 127.31.32.62:44097)
0810 04:07:30.106610 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0810 04:07:30.106619 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 04:07:30.106664 (+ 45us) server_negotiation.cc:407] Connection header received
0810 04:07:30.107144 (+ 480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:07:30.107153 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:07:30.107224 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:07:30.107375 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:07:30.108021 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:30.109385 (+ 1364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:30.110182 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:30.110770 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:30.110895 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:30.113246 (+ 2351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:07:30.113251 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:07:30.113251 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:07:30.113268 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:07:30.115367 (+ 2099us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:07:30.115893 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:07:30.115896 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:07:30.115897 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:07:30.116058 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:07:30.116360 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:07:30.116361 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:07:30.116362 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:07:30.116727 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:07:30.117367 (+ 640us) server_negotiation.cc:1035] Waiting for connection context
0810 04:07:30.117439 (+ 72us) server_negotiation.cc:299] Negotiation successful
0810 04:07:30.117553 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":116,"threads_started":1}
I20260810 04:07:30.118286 10658 catalog_manager.cc:1347] Generated new cluster ID: 75174220094e418597f574a1e3b10ead
I20260810 04:07:30.118367 10658 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 04:07:30.131038 10658 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 04:07:30.131793 10658 catalog_manager.cc:1504] Loading token signing keys...
I20260810 04:07:30.139292 10658 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 416c03f2ba5545a3b640aec05f1027ac: Generated new TSK 0
I20260810 04:07:30.139463 10658 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.32.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.32.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.32.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.1.keytab.
Entry for principal kudu/127.31.32.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.1.keytab.
I20260810 04:07:30.172302 31872 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskws89Io/build/release/bin/kudu
/tmp/dist-test-taskws89Io/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786334794676589-31872-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786334794676589-31872-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786334794676589-31872-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786334794676589-31872-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.32.1:0
--local_ip_for_outbound_sockets=127.31.32.1
--webserver_interface=127.31.32.1
--webserver_port=0
--tserver_master_addrs=127.31.32.62:44097
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.1.keytab
--principal=kudu/127.31.32.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.32.20:38741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786334794676589-31872-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786334794676589-31872-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786334794676589-31872-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:07:30.253597 10673 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:07:30.253806 10673 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:07:30.253829 10673 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:07:30.254961 10673 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:07:30.255003 10673 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:07:30.255054 10673 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.32.1
I20260810 04:07:30.256498 10673 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.32.20:38741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786334794676589-31872-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786334794676589-31872-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.1.keytab
--principal=kudu/127.31.32.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.32.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786334794676589-31872-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.32.1
--webserver_port=0
--tserver_master_addrs=127.31.32.62:44097
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10673
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.32.1
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786334794676589-31872-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:07:30.256742 10673 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:07:30.256974 10673 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:07:30.261966 10679 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:07:30.262097 10673 server_base.cc:1034] running on GCE node
W20260810 04:07:30.261984 10678 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:07:30.261984 10681 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:07:30.262451 10673 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:07:30.262667 10673 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:07:30.263799 10673 hybrid_clock.cc:648] HybridClock initialized: now 1786334850263789 us; error 28 us; skew 500 ppm
Aug 10 04:07:30 dist-test-slave-8w9v krb5kdc[10552](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334850, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:07:30.266351 10673 init.cc:377] Logged in from keytab as kudu/127.31.32.1@KRBTEST.COM (short username kudu)
I20260810 04:07:30.266706 10673 webserver.cc:460] Webserver started at http://127.31.32.1:41883/ using document root <none> and password file <none>
I20260810 04:07:30.266891 10673 fs_manager.cc:359] Metadata directory not provided
I20260810 04:07:30.266932 10673 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:07:30.267096 10673 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:07:30.268079 10673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786334794676589-31872-0/minicluster-data/ts-0/data/instance:
uuid: "9e848cbf7fcd4891aee8f3ebecce1467"
format_stamp: "Formatted at 2026-08-10 04:07:30 on dist-test-slave-8w9v"
server_key: "78063fc6756dbd615853ddaa1921a787"
server_key_iv: "db5be792d4e07196f2474a2b0bcabb78"
server_key_version: "encryptionkey@0"
I20260810 04:07:30.268460 10673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786334794676589-31872-0/minicluster-data/ts-0/wal/instance:
uuid: "9e848cbf7fcd4891aee8f3ebecce1467"
format_stamp: "Formatted at 2026-08-10 04:07:30 on dist-test-slave-8w9v"
server_key: "78063fc6756dbd615853ddaa1921a787"
server_key_iv: "db5be792d4e07196f2474a2b0bcabb78"
server_key_version: "encryptionkey@0"
I20260810 04:07:30.269709 10673 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:07:30.270392 10688 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:07:30.270570 10673 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 04:07:30.270656 10673 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786334794676589-31872-0/minicluster-data/ts-0/data,/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786334794676589-31872-0/minicluster-data/ts-0/wal
uuid: "9e848cbf7fcd4891aee8f3ebecce1467"
format_stamp: "Formatted at 2026-08-10 04:07:30 on dist-test-slave-8w9v"
server_key: "78063fc6756dbd615853ddaa1921a787"
server_key_iv: "db5be792d4e07196f2474a2b0bcabb78"
server_key_version: "encryptionkey@0"
I20260810 04:07:30.270726 10673 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786334794676589-31872-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786334794676589-31872-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786334794676589-31872-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:07:30.281085 10673 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:07:30.283277 10673 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:07:30.283427 10673 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:07:30.283630 10673 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:07:30.283964 10673 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:07:30.283998 10673 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:07:30.284019 10673 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:07:30.284052 10673 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:07:30.289553 10673 rpc_server.cc:307] RPC server started. Bound to: 127.31.32.1:36381
I20260810 04:07:30.289628 10812 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.32.1:36381 every 8 connection(s)
I20260810 04:07:30.289917 10673 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786334794676589-31872-0/minicluster-data/ts-0/data/info.pb
Aug 10 04:07:30 dist-test-slave-8w9v krb5kdc[10552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334850, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.1@KRBTEST.COM for kudu/127.31.32.62@KRBTEST.COM
I20260810 04:07:30.297544 31872 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskws89Io/build/release/bin/kudu as pid 10673
I20260810 04:07:30.297631 31872 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786334794676589-31872-0/minicluster-data/ts-0/wal/instance
I20260810 04:07:30.297863 31872 external_mini_cluster.cc:1428] Setting key 522c15ec5f47974b7279f780330b8dad
I20260810 04:07:30.299121 10815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:30.290630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:44097 (local address 127.31.32.1:37951)
0810 04:07:30.290808 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0810 04:07:30.290860 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0810 04:07:30.291360 (+ 500us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:07:30.291607 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:07:30.291610 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:07:30.292220 (+ 610us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:07:30.292409 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:30.292415 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:30.293153 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:30.293154 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:30.293894 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:30.293896 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:30.294303 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:30.294304 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:30.294425 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:30.294924 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:07:30.294931 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:07:30.296175 (+ 1244us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:07:30.297498 (+ 1323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:07:30.297504 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:07:30.297506 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:07:30.297793 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:07:30.298017 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:07:30.298018 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:07:30.298018 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:07:30.298171 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:07:30.298664 (+ 493us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:07:30.298669 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:07:30.298776 (+ 107us) client_negotiation.cc:769] Sending connection context
0810 04:07:30.298876 (+ 100us) client_negotiation.cc:240] Negotiation successful
0810 04:07:30.298982 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":81,"threads_started":1}
I20260810 04:07:30.299140 10648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:30.290658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.1:37951 (local address 127.31.32.62:44097)
0810 04:07:30.290699 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0810 04:07:30.290701 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:07:30.290937 (+ 236us) server_negotiation.cc:407] Connection header received
0810 04:07:30.291475 (+ 538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:07:30.291476 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:07:30.291488 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:07:30.291537 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:07:30.292517 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:30.293056 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:30.293995 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:30.294199 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:30.294290 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:30.296259 (+ 1969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:07:30.296260 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:07:30.296261 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:07:30.296267 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:07:30.297408 (+ 1141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:07:30.297878 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:07:30.297880 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:07:30.297880 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:07:30.297946 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:07:30.298437 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:07:30.298438 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:07:30.298438 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:07:30.298580 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:07:30.298669 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0810 04:07:30.298919 (+ 250us) server_negotiation.cc:299] Negotiation successful
0810 04:07:30.299017 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260810 04:07:30.300580 10813 heartbeater.cc:347] Connected to a master server at 127.31.32.62:44097
I20260810 04:07:30.300683 10813 heartbeater.cc:464] Registering TS with master...
I20260810 04:07:30.300869 10813 heartbeater.cc:511] Master 127.31.32.62:44097 requested a full tablet report, sending...
I20260810 04:07:30.301314 10603 ts_manager.cc:194] Registered new tserver with Master: 9e848cbf7fcd4891aee8f3ebecce1467 (127.31.32.1:36381)
I20260810 04:07:30.301927 10603 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.32.1@KRBTEST.COM'} at 127.31.32.1:37951
WARNING: no policy specified for kudu/127.31.32.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.32.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.32.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.2.keytab.
Entry for principal kudu/127.31.32.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.2.keytab.
I20260810 04:07:30.323199 31872 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskws89Io/build/release/bin/kudu
/tmp/dist-test-taskws89Io/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786334794676589-31872-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786334794676589-31872-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786334794676589-31872-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786334794676589-31872-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.32.2:0
--local_ip_for_outbound_sockets=127.31.32.2
--webserver_interface=127.31.32.2
--webserver_port=0
--tserver_master_addrs=127.31.32.62:44097
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.2.keytab
--principal=kudu/127.31.32.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.32.20:38741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786334794676589-31872-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786334794676589-31872-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786334794676589-31872-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:07:30.410403 10818 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:07:30.410597 10818 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:07:30.410621 10818 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:07:30.411712 10818 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:07:30.411760 10818 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:07:30.411813 10818 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.32.2
I20260810 04:07:30.413307 10818 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.32.20:38741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786334794676589-31872-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786334794676589-31872-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.2.keytab
--principal=kudu/127.31.32.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.32.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786334794676589-31872-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.32.2
--webserver_port=0
--tserver_master_addrs=127.31.32.62:44097
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10818
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.32.2
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786334794676589-31872-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:07:30.413544 10818 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:07:30.413774 10818 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:07:30.418684 10826 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:07:30.418727 10823 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:07:30.418673 10824 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:07:30.419016 10818 server_base.cc:1034] running on GCE node
I20260810 04:07:30.419217 10818 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:07:30.419467 10818 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:07:30.420611 10818 hybrid_clock.cc:648] HybridClock initialized: now 1786334850420595 us; error 28 us; skew 500 ppm
Aug 10 04:07:30 dist-test-slave-8w9v krb5kdc[10552](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334850, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:07:30.423064 10818 init.cc:377] Logged in from keytab as kudu/127.31.32.2@KRBTEST.COM (short username kudu)
I20260810 04:07:30.423535 10818 webserver.cc:460] Webserver started at http://127.31.32.2:43083/ using document root <none> and password file <none>
I20260810 04:07:30.423758 10818 fs_manager.cc:359] Metadata directory not provided
I20260810 04:07:30.423832 10818 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:07:30.423977 10818 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:07:30.425024 10818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786334794676589-31872-0/minicluster-data/ts-1/data/instance:
uuid: "cf4332b416154c45bbcef6fbb2761601"
format_stamp: "Formatted at 2026-08-10 04:07:30 on dist-test-slave-8w9v"
server_key: "9cf609a333cc78764d279adf6d1b4a85"
server_key_iv: "e93c8c09e940325d02e72b6ff8c352c9"
server_key_version: "encryptionkey@0"
I20260810 04:07:30.425382 10818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786334794676589-31872-0/minicluster-data/ts-1/wal/instance:
uuid: "cf4332b416154c45bbcef6fbb2761601"
format_stamp: "Formatted at 2026-08-10 04:07:30 on dist-test-slave-8w9v"
server_key: "9cf609a333cc78764d279adf6d1b4a85"
server_key_iv: "e93c8c09e940325d02e72b6ff8c352c9"
server_key_version: "encryptionkey@0"
I20260810 04:07:30.426695 10818 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 04:07:30.427423 10834 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:07:30.427619 10818 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 04:07:30.427716 10818 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786334794676589-31872-0/minicluster-data/ts-1/data,/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786334794676589-31872-0/minicluster-data/ts-1/wal
uuid: "cf4332b416154c45bbcef6fbb2761601"
format_stamp: "Formatted at 2026-08-10 04:07:30 on dist-test-slave-8w9v"
server_key: "9cf609a333cc78764d279adf6d1b4a85"
server_key_iv: "e93c8c09e940325d02e72b6ff8c352c9"
server_key_version: "encryptionkey@0"
I20260810 04:07:30.427798 10818 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786334794676589-31872-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786334794676589-31872-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786334794676589-31872-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:07:30.442485 10818 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:07:30.444584 10818 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:07:30.444729 10818 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:07:30.444932 10818 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:07:30.445217 10818 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:07:30.445271 10818 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:07:30.445309 10818 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:07:30.445354 10818 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:07:30.451114 10818 rpc_server.cc:307] RPC server started. Bound to: 127.31.32.2:43497
I20260810 04:07:30.451166 10963 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.32.2:43497 every 8 connection(s)
I20260810 04:07:30.451442 10818 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786334794676589-31872-0/minicluster-data/ts-1/data/info.pb
Aug 10 04:07:30 dist-test-slave-8w9v krb5kdc[10552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334850, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.2@KRBTEST.COM for kudu/127.31.32.62@KRBTEST.COM
I20260810 04:07:30.458114 31872 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskws89Io/build/release/bin/kudu as pid 10818
I20260810 04:07:30.458225 31872 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786334794676589-31872-0/minicluster-data/ts-1/wal/instance
I20260810 04:07:30.458499 31872 external_mini_cluster.cc:1428] Setting key b6dc238919e6525c670db0f5473160af
I20260810 04:07:30.460542 10648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:30.452103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.2:56425 (local address 127.31.32.62:44097)
0810 04:07:30.452169 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0810 04:07:30.452171 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:07:30.452370 (+ 199us) server_negotiation.cc:407] Connection header received
0810 04:07:30.452847 (+ 477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:07:30.452848 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:07:30.452862 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:07:30.452911 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:07:30.453808 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:30.454305 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:30.455246 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:30.455442 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:30.455523 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:30.457546 (+ 2023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:07:30.457547 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:07:30.457547 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:07:30.457556 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:07:30.459117 (+ 1561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:07:30.459591 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:07:30.459592 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:07:30.459592 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:07:30.459643 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:07:30.459925 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:07:30.459926 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:07:30.459926 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:07:30.460026 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:07:30.460078 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0810 04:07:30.460397 (+ 319us) server_negotiation.cc:299] Negotiation successful
0810 04:07:30.460479 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260810 04:07:30.460817 10966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:30.452049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:44097 (local address 127.31.32.2:56425)
0810 04:07:30.452217 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0810 04:07:30.452292 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0810 04:07:30.452753 (+ 461us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:07:30.452987 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:07:30.452991 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:07:30.453516 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:07:30.453696 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:30.453700 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:30.454403 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:30.454404 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:30.455158 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:30.455160 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:30.455534 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:30.455535 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:30.455659 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:30.456036 (+ 377us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:07:30.456043 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:07:30.457450 (+ 1407us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:07:30.459209 (+ 1759us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:07:30.459216 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:07:30.459218 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:07:30.459524 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:07:30.459702 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:07:30.459703 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:07:30.459703 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:07:30.459861 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:07:30.460085 (+ 224us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:07:30.460091 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:07:30.460257 (+ 166us) client_negotiation.cc:769] Sending connection context
0810 04:07:30.460362 (+ 105us) client_negotiation.cc:240] Negotiation successful
0810 04:07:30.460545 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":91,"threads_started":1}
I20260810 04:07:30.461588 10964 heartbeater.cc:347] Connected to a master server at 127.31.32.62:44097
I20260810 04:07:30.461694 10964 heartbeater.cc:464] Registering TS with master...
I20260810 04:07:30.461885 10964 heartbeater.cc:511] Master 127.31.32.62:44097 requested a full tablet report, sending...
I20260810 04:07:30.462284 10603 ts_manager.cc:194] Registered new tserver with Master: cf4332b416154c45bbcef6fbb2761601 (127.31.32.2:43497)
I20260810 04:07:30.462647 10603 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.32.2@KRBTEST.COM'} at 127.31.32.2:56425
WARNING: no policy specified for kudu/127.31.32.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.32.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.32.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.3.keytab.
Entry for principal kudu/127.31.32.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.3.keytab.
I20260810 04:07:30.487941 31872 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskws89Io/build/release/bin/kudu
/tmp/dist-test-taskws89Io/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786334794676589-31872-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786334794676589-31872-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786334794676589-31872-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786334794676589-31872-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.32.3:0
--local_ip_for_outbound_sockets=127.31.32.3
--webserver_interface=127.31.32.3
--webserver_port=0
--tserver_master_addrs=127.31.32.62:44097
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.3.keytab
--principal=kudu/127.31.32.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.32.20:38741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786334794676589-31872-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786334794676589-31872-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786334794676589-31872-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:07:30.567582 10969 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:07:30.567809 10969 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:07:30.567848 10969 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:07:30.569024 10969 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:07:30.569088 10969 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:07:30.569133 10969 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.32.3
I20260810 04:07:30.570575 10969 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.32.20:38741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786334794676589-31872-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786334794676589-31872-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.3.keytab
--principal=kudu/127.31.32.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.32.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786334794676589-31872-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.32.3
--webserver_port=0
--tserver_master_addrs=127.31.32.62:44097
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10969
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.32.3
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786334794676589-31872-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:07:30.570833 10969 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:07:30.571049 10969 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:07:30.576171 10975 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:07:30.576171 10977 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:07:30.576212 10974 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:07:30.576359 10969 server_base.cc:1034] running on GCE node
I20260810 04:07:30.576584 10969 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:07:30.576764 10969 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:07:30.577901 10969 hybrid_clock.cc:648] HybridClock initialized: now 1786334850577896 us; error 35 us; skew 500 ppm
Aug 10 04:07:30 dist-test-slave-8w9v krb5kdc[10552](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334850, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:07:30.580365 10969 init.cc:377] Logged in from keytab as kudu/127.31.32.3@KRBTEST.COM (short username kudu)
I20260810 04:07:30.580811 10969 webserver.cc:460] Webserver started at http://127.31.32.3:39187/ using document root <none> and password file <none>
I20260810 04:07:30.580991 10969 fs_manager.cc:359] Metadata directory not provided
I20260810 04:07:30.581032 10969 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:07:30.581130 10969 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:07:30.582217 10969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786334794676589-31872-0/minicluster-data/ts-2/data/instance:
uuid: "1995cd2f0a934e878ea3afa1081c932a"
format_stamp: "Formatted at 2026-08-10 04:07:30 on dist-test-slave-8w9v"
server_key: "811ddc0bd51441349388f4bbdb1c6295"
server_key_iv: "b3590eabca6daf55c47f197ab411fa44"
server_key_version: "encryptionkey@0"
I20260810 04:07:30.582525 10969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786334794676589-31872-0/minicluster-data/ts-2/wal/instance:
uuid: "1995cd2f0a934e878ea3afa1081c932a"
format_stamp: "Formatted at 2026-08-10 04:07:30 on dist-test-slave-8w9v"
server_key: "811ddc0bd51441349388f4bbdb1c6295"
server_key_iv: "b3590eabca6daf55c47f197ab411fa44"
server_key_version: "encryptionkey@0"
I20260810 04:07:30.583988 10969 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 04:07:30.584797 10984 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:07:30.584975 10969 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 04:07:30.585067 10969 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786334794676589-31872-0/minicluster-data/ts-2/data,/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786334794676589-31872-0/minicluster-data/ts-2/wal
uuid: "1995cd2f0a934e878ea3afa1081c932a"
format_stamp: "Formatted at 2026-08-10 04:07:30 on dist-test-slave-8w9v"
server_key: "811ddc0bd51441349388f4bbdb1c6295"
server_key_iv: "b3590eabca6daf55c47f197ab411fa44"
server_key_version: "encryptionkey@0"
I20260810 04:07:30.585140 10969 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786334794676589-31872-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786334794676589-31872-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786334794676589-31872-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:07:30.608428 10969 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:07:30.610283 10969 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:07:30.610437 10969 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:07:30.610646 10969 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:07:30.610946 10969 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:07:30.610998 10969 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:07:30.611030 10969 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:07:30.611060 10969 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:07:30.617260 10969 rpc_server.cc:307] RPC server started. Bound to: 127.31.32.3:46367
I20260810 04:07:30.617314 11112 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.32.3:46367 every 8 connection(s)
I20260810 04:07:30.617628 10969 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786334794676589-31872-0/minicluster-data/ts-2/data/info.pb
I20260810 04:07:30.622412 31872 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskws89Io/build/release/bin/kudu as pid 10969
I20260810 04:07:30.622480 31872 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786334794676589-31872-0/minicluster-data/ts-2/wal/instance
I20260810 04:07:30.622670 31872 external_mini_cluster.cc:1428] Setting key ab37f621ff3e6b1eb9a2de91f13648bf
Aug 10 04:07:30 dist-test-slave-8w9v krb5kdc[10552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334850, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.3@KRBTEST.COM for kudu/127.31.32.62@KRBTEST.COM
I20260810 04:07:30.626211 10648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:30.618321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.3:41011 (local address 127.31.32.62:44097)
0810 04:07:30.618378 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0810 04:07:30.618380 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:07:30.618548 (+ 168us) server_negotiation.cc:407] Connection header received
0810 04:07:30.619053 (+ 505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:07:30.619054 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:07:30.619064 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:07:30.619116 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:07:30.619995 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:30.620546 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:30.621505 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:30.621685 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:30.621748 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:30.623970 (+ 2222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:07:30.623972 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:07:30.623972 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:07:30.623980 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:07:30.625186 (+ 1206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:07:30.625520 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:07:30.625521 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:07:30.625521 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:07:30.625569 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:07:30.625766 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:07:30.625767 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:07:30.625767 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:07:30.625858 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:07:30.625906 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0810 04:07:30.626095 (+ 189us) server_negotiation.cc:299] Negotiation successful
0810 04:07:30.626150 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260810 04:07:30.626430 11116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:30.618309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:44097 (local address 127.31.32.3:41011)
0810 04:07:30.618451 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0810 04:07:30.618483 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0810 04:07:30.618967 (+ 484us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:07:30.619169 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:07:30.619172 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:07:30.619726 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:07:30.619923 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:30.619928 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:30.620629 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:30.620630 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:30.621437 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:30.621438 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:30.621780 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:30.621781 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:30.621891 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:30.622373 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:07:30.622381 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:07:30.623891 (+ 1510us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:07:30.625249 (+ 1358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:07:30.625254 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:07:30.625256 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:07:30.625459 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:07:30.625620 (+ 161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:07:30.625621 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:07:30.625621 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:07:30.625717 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:07:30.625907 (+ 190us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:07:30.625910 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:07:30.626009 (+ 99us) client_negotiation.cc:769] Sending connection context
0810 04:07:30.626084 (+ 75us) client_negotiation.cc:240] Negotiation successful
0810 04:07:30.626190 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":76,"threads_started":1}
I20260810 04:07:30.626863 11113 heartbeater.cc:347] Connected to a master server at 127.31.32.62:44097
I20260810 04:07:30.626955 11113 heartbeater.cc:464] Registering TS with master...
I20260810 04:07:30.627130 11113 heartbeater.cc:511] Master 127.31.32.62:44097 requested a full tablet report, sending...
I20260810 04:07:30.627507 10603 ts_manager.cc:194] Registered new tserver with Master: 1995cd2f0a934e878ea3afa1081c932a (127.31.32.3:46367)
I20260810 04:07:30.627858 10603 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.32.3@KRBTEST.COM'} at 127.31.32.3:41011
I20260810 04:07:30.634991 31872 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 04:07:30.641815 10648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:30.636278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39560 (local address 127.31.32.62:44097)
0810 04:07:30.636345 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0810 04:07:30.636346 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 04:07:30.636429 (+ 83us) server_negotiation.cc:407] Connection header received
0810 04:07:30.636463 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:07:30.636464 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:07:30.636472 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:07:30.636520 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:07:30.637005 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:30.637451 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:30.638103 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:30.638269 (+ 166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:30.638327 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:30.639457 (+ 1130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:07:30.639459 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:07:30.639459 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:07:30.639466 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:07:30.640682 (+ 1216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:07:30.641010 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:07:30.641011 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:07:30.641011 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:07:30.641084 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:07:30.641280 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:07:30.641281 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:07:30.641282 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:07:30.641430 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:07:30.641535 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0810 04:07:30.641649 (+ 114us) server_negotiation.cc:299] Negotiation successful
0810 04:07:30.641720 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260810 04:07:30.643316 10603 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39560:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 04:07:30.644970 10603 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 04:07:30.651101 11131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:30.647340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40912 (local address 127.31.32.1:36381)
0810 04:07:30.647757 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0810 04:07:30.647761 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 04:07:30.647771 (+ 10us) server_negotiation.cc:407] Connection header received
0810 04:07:30.647808 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:07:30.647811 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:07:30.647824 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:07:30.648033 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:07:30.648111 (+ 78us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:30.649163 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:30.650178 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:30.650712 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:30.650793 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:30.650857 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0810 04:07:30.650971 (+ 114us) server_negotiation.cc:299] Negotiation successful
0810 04:07:30.651010 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":79,"threads_started":1}
I20260810 04:07:30.651131 11130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:30.647130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.1:36381 (local address 127.0.0.1:40912)
0810 04:07:30.647570 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0810 04:07:30.647581 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 04:07:30.647638 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:07:30.647916 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:07:30.647917 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:07:30.647918 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:07:30.647961 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:30.647963 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:30.649280 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:30.649281 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:30.650101 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:30.650104 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:30.650794 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:30.650795 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:30.650873 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:30.650886 (+ 13us) client_negotiation.cc:769] Sending connection context
0810 04:07:30.650957 (+ 71us) client_negotiation.cc:240] Negotiation successful
0810 04:07:30.651006 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":405,"mutex_wait_us":2,"thread_start_us":62,"threads_started":1}
I20260810 04:07:30.651139 11127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:30.647327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.3:46367 (local address 127.0.0.1:40174)
0810 04:07:30.647670 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0810 04:07:30.647677 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0810 04:07:30.647729 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:07:30.647963 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:07:30.647964 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:07:30.647966 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:07:30.648020 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:30.648021 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:30.648872 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:30.648873 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:30.649999 (+ 1126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:30.650002 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:30.650866 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:30.650868 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:30.650929 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:30.650932 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 04:07:30.650983 (+ 51us) client_negotiation.cc:240] Negotiation successful
0810 04:07:30.651028 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":74,"threads_started":1}
I20260810 04:07:30.651304 11132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:30.647419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40174 (local address 127.31.32.3:46367)
0810 04:07:30.647564 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0810 04:07:30.647569 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 04:07:30.647727 (+ 158us) server_negotiation.cc:407] Connection header received
0810 04:07:30.647788 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:07:30.647790 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:07:30.647803 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:07:30.647887 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:07:30.648096 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:30.648769 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:30.650070 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:30.650715 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:30.650780 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:30.650830 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0810 04:07:30.651175 (+ 345us) server_negotiation.cc:299] Negotiation successful
0810 04:07:30.651216 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":71,"threads_started":1}
I20260810 04:07:30.651401 11128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:30.647130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.2:43497 (local address 127.0.0.1:59248)
0810 04:07:30.647462 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0810 04:07:30.647498 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0810 04:07:30.647578 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:07:30.647857 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:07:30.647860 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:07:30.647862 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:07:30.647922 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:30.647923 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:30.649234 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:30.649235 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:30.650297 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:30.650299 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:30.651178 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:30.651179 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:30.651233 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:30.651236 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 04:07:30.651289 (+ 53us) client_negotiation.cc:240] Negotiation successful
0810 04:07:30.651329 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":54,"threads_started":1}
I20260810 04:07:30.651461 11129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:30.647236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59248 (local address 127.31.32.2:43497)
0810 04:07:30.647433 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0810 04:07:30.647438 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 04:07:30.647579 (+ 141us) server_negotiation.cc:407] Connection header received
0810 04:07:30.647660 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:07:30.647662 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:07:30.647678 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:07:30.647755 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:07:30.648001 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:30.649127 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:30.650375 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:30.651040 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:30.651098 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:30.651144 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0810 04:07:30.651314 (+ 170us) server_negotiation.cc:299] Negotiation successful
0810 04:07:30.651364 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":86,"threads_started":1}
I20260810 04:07:30.652113 10889 tablet_service.cc:1467] Processing CreateTablet for tablet 3226a24d4c4146b985ad923c3ffaaee0 (DEFAULT_TABLE table=test-table [id=526e9552e184418ebc9120c8c341e15e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:07:30.652110 11037 tablet_service.cc:1467] Processing CreateTablet for tablet 3226a24d4c4146b985ad923c3ffaaee0 (DEFAULT_TABLE table=test-table [id=526e9552e184418ebc9120c8c341e15e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:07:30.652386 10740 tablet_service.cc:1467] Processing CreateTablet for tablet 3226a24d4c4146b985ad923c3ffaaee0 (DEFAULT_TABLE table=test-table [id=526e9552e184418ebc9120c8c341e15e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:07:30.652506 10889 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3226a24d4c4146b985ad923c3ffaaee0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:07:30.652506 11037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3226a24d4c4146b985ad923c3ffaaee0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:07:30.652731 10740 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3226a24d4c4146b985ad923c3ffaaee0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:07:30.655050 11133 tablet_bootstrap.cc:492] T 3226a24d4c4146b985ad923c3ffaaee0 P 9e848cbf7fcd4891aee8f3ebecce1467: Bootstrap starting.
I20260810 04:07:30.655711 11133 tablet_bootstrap.cc:654] T 3226a24d4c4146b985ad923c3ffaaee0 P 9e848cbf7fcd4891aee8f3ebecce1467: Neither blocks nor log segments found. Creating new log.
I20260810 04:07:30.655740 11134 tablet_bootstrap.cc:492] T 3226a24d4c4146b985ad923c3ffaaee0 P 1995cd2f0a934e878ea3afa1081c932a: Bootstrap starting.
I20260810 04:07:30.656266 11135 tablet_bootstrap.cc:492] T 3226a24d4c4146b985ad923c3ffaaee0 P cf4332b416154c45bbcef6fbb2761601: Bootstrap starting.
I20260810 04:07:30.656440 11133 log.cc:824] T 3226a24d4c4146b985ad923c3ffaaee0 P 9e848cbf7fcd4891aee8f3ebecce1467: Log is configured to *not* fsync() on all Append() calls
I20260810 04:07:30.656620 11134 tablet_bootstrap.cc:654] T 3226a24d4c4146b985ad923c3ffaaee0 P 1995cd2f0a934e878ea3afa1081c932a: Neither blocks nor log segments found. Creating new log.
I20260810 04:07:30.656930 11135 tablet_bootstrap.cc:654] T 3226a24d4c4146b985ad923c3ffaaee0 P cf4332b416154c45bbcef6fbb2761601: Neither blocks nor log segments found. Creating new log.
I20260810 04:07:30.657128 11134 log.cc:824] T 3226a24d4c4146b985ad923c3ffaaee0 P 1995cd2f0a934e878ea3afa1081c932a: Log is configured to *not* fsync() on all Append() calls
I20260810 04:07:30.657305 11133 tablet_bootstrap.cc:492] T 3226a24d4c4146b985ad923c3ffaaee0 P 9e848cbf7fcd4891aee8f3ebecce1467: No bootstrap required, opened a new log
I20260810 04:07:30.657387 11133 ts_tablet_manager.cc:1397] T 3226a24d4c4146b985ad923c3ffaaee0 P 9e848cbf7fcd4891aee8f3ebecce1467: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 04:07:30.657505 11135 log.cc:824] T 3226a24d4c4146b985ad923c3ffaaee0 P cf4332b416154c45bbcef6fbb2761601: Log is configured to *not* fsync() on all Append() calls
I20260810 04:07:30.657971 11134 tablet_bootstrap.cc:492] T 3226a24d4c4146b985ad923c3ffaaee0 P 1995cd2f0a934e878ea3afa1081c932a: No bootstrap required, opened a new log
I20260810 04:07:30.658041 11134 ts_tablet_manager.cc:1397] T 3226a24d4c4146b985ad923c3ffaaee0 P 1995cd2f0a934e878ea3afa1081c932a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 04:07:30.658182 11135 tablet_bootstrap.cc:492] T 3226a24d4c4146b985ad923c3ffaaee0 P cf4332b416154c45bbcef6fbb2761601: No bootstrap required, opened a new log
I20260810 04:07:30.658258 11135 ts_tablet_manager.cc:1397] T 3226a24d4c4146b985ad923c3ffaaee0 P cf4332b416154c45bbcef6fbb2761601: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 04:07:30.659317 11133 raft_consensus.cc:348] T 3226a24d4c4146b985ad923c3ffaaee0 P 9e848cbf7fcd4891aee8f3ebecce1467 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e848cbf7fcd4891aee8f3ebecce1467" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 36381 } } peers { permanent_uuid: "cf4332b416154c45bbcef6fbb2761601" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 43497 } } peers { permanent_uuid: "1995cd2f0a934e878ea3afa1081c932a" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 46367 } }
I20260810 04:07:30.659495 11133 raft_consensus.cc:374] T 3226a24d4c4146b985ad923c3ffaaee0 P 9e848cbf7fcd4891aee8f3ebecce1467 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:07:30.659530 11133 raft_consensus.cc:729] T 3226a24d4c4146b985ad923c3ffaaee0 P 9e848cbf7fcd4891aee8f3ebecce1467 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e848cbf7fcd4891aee8f3ebecce1467, State: Initialized, Role: FOLLOWER
I20260810 04:07:30.659683 11133 consensus_queue.cc:260] T 3226a24d4c4146b985ad923c3ffaaee0 P 9e848cbf7fcd4891aee8f3ebecce1467 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e848cbf7fcd4891aee8f3ebecce1467" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 36381 } } peers { permanent_uuid: "cf4332b416154c45bbcef6fbb2761601" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 43497 } } peers { permanent_uuid: "1995cd2f0a934e878ea3afa1081c932a" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 46367 } }
I20260810 04:07:30.659961 11133 ts_tablet_manager.cc:1428] T 3226a24d4c4146b985ad923c3ffaaee0 P 9e848cbf7fcd4891aee8f3ebecce1467: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 04:07:30.659977 10813 heartbeater.cc:503] Master 127.31.32.62:44097 was elected leader, sending a full tablet report...
I20260810 04:07:30.659894 11134 raft_consensus.cc:348] T 3226a24d4c4146b985ad923c3ffaaee0 P 1995cd2f0a934e878ea3afa1081c932a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e848cbf7fcd4891aee8f3ebecce1467" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 36381 } } peers { permanent_uuid: "cf4332b416154c45bbcef6fbb2761601" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 43497 } } peers { permanent_uuid: "1995cd2f0a934e878ea3afa1081c932a" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 46367 } }
I20260810 04:07:30.659919 11135 raft_consensus.cc:348] T 3226a24d4c4146b985ad923c3ffaaee0 P cf4332b416154c45bbcef6fbb2761601 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e848cbf7fcd4891aee8f3ebecce1467" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 36381 } } peers { permanent_uuid: "cf4332b416154c45bbcef6fbb2761601" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 43497 } } peers { permanent_uuid: "1995cd2f0a934e878ea3afa1081c932a" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 46367 } }
I20260810 04:07:30.660039 11134 raft_consensus.cc:374] T 3226a24d4c4146b985ad923c3ffaaee0 P 1995cd2f0a934e878ea3afa1081c932a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:07:30.660076 11134 raft_consensus.cc:729] T 3226a24d4c4146b985ad923c3ffaaee0 P 1995cd2f0a934e878ea3afa1081c932a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1995cd2f0a934e878ea3afa1081c932a, State: Initialized, Role: FOLLOWER
I20260810 04:07:30.660066 11135 raft_consensus.cc:374] T 3226a24d4c4146b985ad923c3ffaaee0 P cf4332b416154c45bbcef6fbb2761601 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:07:30.660099 11135 raft_consensus.cc:729] T 3226a24d4c4146b985ad923c3ffaaee0 P cf4332b416154c45bbcef6fbb2761601 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf4332b416154c45bbcef6fbb2761601, State: Initialized, Role: FOLLOWER
I20260810 04:07:30.660230 11134 consensus_queue.cc:260] T 3226a24d4c4146b985ad923c3ffaaee0 P 1995cd2f0a934e878ea3afa1081c932a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e848cbf7fcd4891aee8f3ebecce1467" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 36381 } } peers { permanent_uuid: "cf4332b416154c45bbcef6fbb2761601" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 43497 } } peers { permanent_uuid: "1995cd2f0a934e878ea3afa1081c932a" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 46367 } }
I20260810 04:07:30.660259 11135 consensus_queue.cc:260] T 3226a24d4c4146b985ad923c3ffaaee0 P cf4332b416154c45bbcef6fbb2761601 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e848cbf7fcd4891aee8f3ebecce1467" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 36381 } } peers { permanent_uuid: "cf4332b416154c45bbcef6fbb2761601" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 43497 } } peers { permanent_uuid: "1995cd2f0a934e878ea3afa1081c932a" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 46367 } }
I20260810 04:07:30.660523 11135 ts_tablet_manager.cc:1428] T 3226a24d4c4146b985ad923c3ffaaee0 P cf4332b416154c45bbcef6fbb2761601: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260810 04:07:30.660526 11113 heartbeater.cc:503] Master 127.31.32.62:44097 was elected leader, sending a full tablet report...
I20260810 04:07:30.660748 10964 heartbeater.cc:503] Master 127.31.32.62:44097 was elected leader, sending a full tablet report...
I20260810 04:07:30.660995 11134 ts_tablet_manager.cc:1428] T 3226a24d4c4146b985ad923c3ffaaee0 P 1995cd2f0a934e878ea3afa1081c932a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
W20260810 04:07:30.701924 10965 tablet.cc:2369] T 3226a24d4c4146b985ad923c3ffaaee0 P cf4332b416154c45bbcef6fbb2761601: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 04:07:30.704270 11139 raft_consensus.cc:482] T 3226a24d4c4146b985ad923c3ffaaee0 P 9e848cbf7fcd4891aee8f3ebecce1467 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 04:07:30.704494 11139 raft_consensus.cc:504] T 3226a24d4c4146b985ad923c3ffaaee0 P 9e848cbf7fcd4891aee8f3ebecce1467 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e848cbf7fcd4891aee8f3ebecce1467" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 36381 } } peers { permanent_uuid: "cf4332b416154c45bbcef6fbb2761601" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 43497 } } peers { permanent_uuid: "1995cd2f0a934e878ea3afa1081c932a" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 46367 } }
I20260810 04:07:30.704855 11139 leader_election.cc:288] T 3226a24d4c4146b985ad923c3ffaaee0 P 9e848cbf7fcd4891aee8f3ebecce1467 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cf4332b416154c45bbcef6fbb2761601 (127.31.32.2:43497), 1995cd2f0a934e878ea3afa1081c932a (127.31.32.3:46367)
I20260810 04:07:30.707526 10815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:30.705031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.3:46367 (local address 127.31.32.1:37133)
0810 04:07:30.705090 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0810 04:07:30.705105 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 04:07:30.705158 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:07:30.705311 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:07:30.705333 (+ 22us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:07:30.705335 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:07:30.705420 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:30.705422 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:30.706038 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:30.706040 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:30.706749 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:30.706751 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:30.707273 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:30.707274 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:30.707356 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:30.707359 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 04:07:30.707421 (+ 62us) client_negotiation.cc:240] Negotiation successful
0810 04:07:30.707460 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260810 04:07:30.707526 11132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:30.705121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.1:37133 (local address 127.31.32.3:46367)
0810 04:07:30.705162 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0810 04:07:30.705163 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 04:07:30.705172 (+ 9us) server_negotiation.cc:407] Connection header received
0810 04:07:30.705223 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:07:30.705224 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:07:30.705235 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:07:30.705295 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:07:30.705485 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:30.705951 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:30.706837 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:30.707197 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:30.707275 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:30.707285 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0810 04:07:30.707426 (+ 141us) server_negotiation.cc:299] Negotiation successful
0810 04:07:30.707465 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260810 04:07:30.707782 11129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:30.705124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.1:51733 (local address 127.31.32.2:43497)
0810 04:07:30.705167 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0810 04:07:30.705169 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:07:30.705374 (+ 205us) server_negotiation.cc:407] Connection header received
0810 04:07:30.705522 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:07:30.705523 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:07:30.705534 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:07:30.705599 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:07:30.705749 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:30.706182 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:30.707039 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:30.707407 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:30.707486 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:30.707494 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0810 04:07:30.707669 (+ 175us) server_negotiation.cc:299] Negotiation successful
0810 04:07:30.707715 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260810 04:07:30.707828 11061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3226a24d4c4146b985ad923c3ffaaee0" candidate_uuid: "9e848cbf7fcd4891aee8f3ebecce1467" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1995cd2f0a934e878ea3afa1081c932a" is_pre_election: true
I20260810 04:07:30.708048 11142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:30.705031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.2:43497 (local address 127.31.32.1:51733)
0810 04:07:30.705248 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0810 04:07:30.705257 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 04:07:30.705404 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:07:30.705603 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:07:30.705604 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:07:30.705605 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:07:30.705684 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:30.705686 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:30.706260 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:30.706261 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:30.706955 (+ 694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:07:30.706957 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:07:30.707489 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:07:30.707490 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:07:30.707560 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:30.707563 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 04:07:30.707898 (+ 335us) client_negotiation.cc:240] Negotiation successful
0810 04:07:30.707950 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"mutex_wait_us":49,"thread_start_us":116,"threads_started":1}
I20260810 04:07:30.708039 11061 raft_consensus.cc:2454] T 3226a24d4c4146b985ad923c3ffaaee0 P 1995cd2f0a934e878ea3afa1081c932a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e848cbf7fcd4891aee8f3ebecce1467 in term 0.
I20260810 04:07:30.708289 10691 leader_election.cc:302] T 3226a24d4c4146b985ad923c3ffaaee0 P 9e848cbf7fcd4891aee8f3ebecce1467 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1995cd2f0a934e878ea3afa1081c932a, 9e848cbf7fcd4891aee8f3ebecce1467; no voters:
I20260810 04:07:30.708425 11139 raft_consensus.cc:2781] T 3226a24d4c4146b985ad923c3ffaaee0 P 9e848cbf7fcd4891aee8f3ebecce1467 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 04:07:30.708478 11139 raft_consensus.cc:482] T 3226a24d4c4146b985ad923c3ffaaee0 P 9e848cbf7fcd4891aee8f3ebecce1467 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 04:07:30.708493 11139 raft_consensus.cc:3037] T 3226a24d4c4146b985ad923c3ffaaee0 P 9e848cbf7fcd4891aee8f3ebecce1467 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:07:30.708411 10912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3226a24d4c4146b985ad923c3ffaaee0" candidate_uuid: "9e848cbf7fcd4891aee8f3ebecce1467" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf4332b416154c45bbcef6fbb2761601" is_pre_election: true
I20260810 04:07:30.708570 10912 raft_consensus.cc:2454] T 3226a24d4c4146b985ad923c3ffaaee0 P cf4332b416154c45bbcef6fbb2761601 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e848cbf7fcd4891aee8f3ebecce1467 in term 0.
I20260810 04:07:30.709227 11139 raft_consensus.cc:504] T 3226a24d4c4146b985ad923c3ffaaee0 P 9e848cbf7fcd4891aee8f3ebecce1467 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e848cbf7fcd4891aee8f3ebecce1467" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 36381 } } peers { permanent_uuid: "cf4332b416154c45bbcef6fbb2761601" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 43497 } } peers { permanent_uuid: "1995cd2f0a934e878ea3afa1081c932a" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 46367 } }
I20260810 04:07:30.709357 11139 leader_election.cc:288] T 3226a24d4c4146b985ad923c3ffaaee0 P 9e848cbf7fcd4891aee8f3ebecce1467 [CANDIDATE]: Term 1 election: Requested vote from peers cf4332b416154c45bbcef6fbb2761601 (127.31.32.2:43497), 1995cd2f0a934e878ea3afa1081c932a (127.31.32.3:46367)
I20260810 04:07:30.709506 10912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3226a24d4c4146b985ad923c3ffaaee0" candidate_uuid: "9e848cbf7fcd4891aee8f3ebecce1467" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf4332b416154c45bbcef6fbb2761601"
I20260810 04:07:30.709573 10912 raft_consensus.cc:3037] T 3226a24d4c4146b985ad923c3ffaaee0 P cf4332b416154c45bbcef6fbb2761601 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:07:30.710325 10912 raft_consensus.cc:2454] T 3226a24d4c4146b985ad923c3ffaaee0 P cf4332b416154c45bbcef6fbb2761601 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e848cbf7fcd4891aee8f3ebecce1467 in term 1.
I20260810 04:07:30.710438 11061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3226a24d4c4146b985ad923c3ffaaee0" candidate_uuid: "9e848cbf7fcd4891aee8f3ebecce1467" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1995cd2f0a934e878ea3afa1081c932a"
I20260810 04:07:30.710484 10690 leader_election.cc:302] T 3226a24d4c4146b985ad923c3ffaaee0 P 9e848cbf7fcd4891aee8f3ebecce1467 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9e848cbf7fcd4891aee8f3ebecce1467, cf4332b416154c45bbcef6fbb2761601; no voters:
I20260810 04:07:30.710516 11061 raft_consensus.cc:3037] T 3226a24d4c4146b985ad923c3ffaaee0 P 1995cd2f0a934e878ea3afa1081c932a [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:07:30.710580 11139 raft_consensus.cc:2781] T 3226a24d4c4146b985ad923c3ffaaee0 P 9e848cbf7fcd4891aee8f3ebecce1467 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:07:30.710731 11139 raft_consensus.cc:686] T 3226a24d4c4146b985ad923c3ffaaee0 P 9e848cbf7fcd4891aee8f3ebecce1467 [term 1 LEADER]: Becoming Leader. State: Replica: 9e848cbf7fcd4891aee8f3ebecce1467, State: Running, Role: LEADER
I20260810 04:07:30.710824 11139 consensus_queue.cc:237] T 3226a24d4c4146b985ad923c3ffaaee0 P 9e848cbf7fcd4891aee8f3ebecce1467 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e848cbf7fcd4891aee8f3ebecce1467" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 36381 } } peers { permanent_uuid: "cf4332b416154c45bbcef6fbb2761601" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 43497 } } peers { permanent_uuid: "1995cd2f0a934e878ea3afa1081c932a" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 46367 } }
I20260810 04:07:30.711444 11061 raft_consensus.cc:2454] T 3226a24d4c4146b985ad923c3ffaaee0 P 1995cd2f0a934e878ea3afa1081c932a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e848cbf7fcd4891aee8f3ebecce1467 in term 1.
I20260810 04:07:30.711659 10603 catalog_manager.cc:5495] T 3226a24d4c4146b985ad923c3ffaaee0 P 9e848cbf7fcd4891aee8f3ebecce1467 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9e848cbf7fcd4891aee8f3ebecce1467 (127.31.32.1). New cstate: current_term: 1 leader_uuid: "9e848cbf7fcd4891aee8f3ebecce1467" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e848cbf7fcd4891aee8f3ebecce1467" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 36381 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cf4332b416154c45bbcef6fbb2761601" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 43497 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1995cd2f0a934e878ea3afa1081c932a" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 46367 } health_report { overall_health: UNKNOWN } } }
I20260810 04:07:30.726652 11131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:30.724683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40928 (local address 127.31.32.1:36381)
0810 04:07:30.724723 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0810 04:07:30.724724 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 04:07:30.724734 (+ 10us) server_negotiation.cc:407] Connection header received
0810 04:07:30.724801 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:07:30.724802 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:07:30.724826 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:07:30.724883 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 04:07:30.724990 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:30.725410 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:30.726087 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:07:30.726248 (+ 161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:07:30.726294 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:07:30.726412 (+ 118us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 04:07:30.726494 (+ 82us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 04:07:30.726547 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0810 04:07:30.726576 (+ 29us) server_negotiation.cc:299] Negotiation successful
0810 04:07:30.726589 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260810 04:07:30.728307 11061 raft_consensus.cc:1261] T 3226a24d4c4146b985ad923c3ffaaee0 P 1995cd2f0a934e878ea3afa1081c932a [term 1 FOLLOWER]: Refusing update from remote peer 9e848cbf7fcd4891aee8f3ebecce1467: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:07:30.728307 10912 raft_consensus.cc:1261] T 3226a24d4c4146b985ad923c3ffaaee0 P cf4332b416154c45bbcef6fbb2761601 [term 1 FOLLOWER]: Refusing update from remote peer 9e848cbf7fcd4891aee8f3ebecce1467: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:07:30.728662 11143 consensus_queue.cc:1035] T 3226a24d4c4146b985ad923c3ffaaee0 P 9e848cbf7fcd4891aee8f3ebecce1467 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf4332b416154c45bbcef6fbb2761601" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 43497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:07:30.728775 11139 consensus_queue.cc:1035] T 3226a24d4c4146b985ad923c3ffaaee0 P 9e848cbf7fcd4891aee8f3ebecce1467 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1995cd2f0a934e878ea3afa1081c932a" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 46367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:07:30.731047 11148 mvcc.cc:205] Tried to move back new op lower bound from 7316827548580392960 to 7316827548512194560. Current Snapshot: MvccSnapshot[applied={T|T < 7316827548580392960}]
I20260810 04:07:30.731482 11149 mvcc.cc:205] Tried to move back new op lower bound from 7316827548580392960 to 7316827548512194560. Current Snapshot: MvccSnapshot[applied={T|T < 7316827548580392960}]
I20260810 04:07:30.733227 10603 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39560:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 04:07:30.733315 10603 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39560:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 04:07:30.734010 10603 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 416c03f2ba5545a3b640aec05f1027ac: Sending DeleteTablet for 3 replicas of tablet 3226a24d4c4146b985ad923c3ffaaee0
I20260810 04:07:30.734380 10740 tablet_service.cc:1514] Processing DeleteTablet for tablet 3226a24d4c4146b985ad923c3ffaaee0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:07:30 UTC) from {username='kudu', principal='kudu/127.31.32.62@KRBTEST.COM'} at 127.0.0.1:40912
I20260810 04:07:30.734457 10889 tablet_service.cc:1514] Processing DeleteTablet for tablet 3226a24d4c4146b985ad923c3ffaaee0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:07:30 UTC) from {username='kudu', principal='kudu/127.31.32.62@KRBTEST.COM'} at 127.0.0.1:59248
I20260810 04:07:30.734618 31872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskws89Io/build/release/bin/kudu with pid 10673
I20260810 04:07:30.734658 11037 tablet_service.cc:1514] Processing DeleteTablet for tablet 3226a24d4c4146b985ad923c3ffaaee0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:07:30 UTC) from {username='kudu', principal='kudu/127.31.32.62@KRBTEST.COM'} at 127.0.0.1:40174
I20260810 04:07:30.740247 11159 tablet_replica.cc:330] T 3226a24d4c4146b985ad923c3ffaaee0 P 1995cd2f0a934e878ea3afa1081c932a: stopping tablet replica
I20260810 04:07:30.740365 11159 raft_consensus.cc:2229] T 3226a24d4c4146b985ad923c3ffaaee0 P 1995cd2f0a934e878ea3afa1081c932a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 04:07:30.740435 11157 tablet_replica.cc:330] T 3226a24d4c4146b985ad923c3ffaaee0 P cf4332b416154c45bbcef6fbb2761601: stopping tablet replica
I20260810 04:07:30.740482 11159 raft_consensus.cc:2258] T 3226a24d4c4146b985ad923c3ffaaee0 P 1995cd2f0a934e878ea3afa1081c932a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:07:30.740556 11157 raft_consensus.cc:2229] T 3226a24d4c4146b985ad923c3ffaaee0 P cf4332b416154c45bbcef6fbb2761601 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 04:07:30.740877 11157 raft_consensus.cc:2258] T 3226a24d4c4146b985ad923c3ffaaee0 P cf4332b416154c45bbcef6fbb2761601 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:07:30.740892 11159 ts_tablet_manager.cc:1905] T 3226a24d4c4146b985ad923c3ffaaee0 P 1995cd2f0a934e878ea3afa1081c932a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:07:30.741384 11157 ts_tablet_manager.cc:1905] T 3226a24d4c4146b985ad923c3ffaaee0 P cf4332b416154c45bbcef6fbb2761601: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:07:30.742723 11159 ts_tablet_manager.cc:1918] T 3226a24d4c4146b985ad923c3ffaaee0 P 1995cd2f0a934e878ea3afa1081c932a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 04:07:30.742810 11159 log.cc:1196] T 3226a24d4c4146b985ad923c3ffaaee0 P 1995cd2f0a934e878ea3afa1081c932a: Deleting WAL directory at /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786334794676589-31872-0/minicluster-data/ts-2/wal/wals/3226a24d4c4146b985ad923c3ffaaee0
I20260810 04:07:30.743062 11159 ts_tablet_manager.cc:1939] T 3226a24d4c4146b985ad923c3ffaaee0 P 1995cd2f0a934e878ea3afa1081c932a: Deleting consensus metadata
I20260810 04:07:30.743393 11157 ts_tablet_manager.cc:1918] T 3226a24d4c4146b985ad923c3ffaaee0 P cf4332b416154c45bbcef6fbb2761601: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 04:07:30.743467 11157 log.cc:1196] T 3226a24d4c4146b985ad923c3ffaaee0 P cf4332b416154c45bbcef6fbb2761601: Deleting WAL directory at /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786334794676589-31872-0/minicluster-data/ts-1/wal/wals/3226a24d4c4146b985ad923c3ffaaee0
I20260810 04:07:30.743460 10588 catalog_manager.cc:4841] TS 1995cd2f0a934e878ea3afa1081c932a (127.31.32.3:46367): tablet 3226a24d4c4146b985ad923c3ffaaee0 (table test-table [id=526e9552e184418ebc9120c8c341e15e]) successfully deleted
I20260810 04:07:30.743743 11157 ts_tablet_manager.cc:1939] T 3226a24d4c4146b985ad923c3ffaaee0 P cf4332b416154c45bbcef6fbb2761601: Deleting consensus metadata
I20260810 04:07:30.744210 10587 catalog_manager.cc:4841] TS cf4332b416154c45bbcef6fbb2761601 (127.31.32.2:43497): tablet 3226a24d4c4146b985ad923c3ffaaee0 (table test-table [id=526e9552e184418ebc9120c8c341e15e]) successfully deleted
W20260810 04:07:30.744395 10588 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.32.1:36381 (error 108)
I20260810 04:07:30.744673 31872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskws89Io/build/release/bin/kudu with pid 10818
I20260810 04:07:30.745535 11128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:07:30.744739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.1:36381 (local address 127.0.0.1:40942)
0810 04:07:30.745459 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0810 04:07:30.745496 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.1:36381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":702}
W20260810 04:07:30.747715 10588 catalog_manager.cc:4584] TS 9e848cbf7fcd4891aee8f3ebecce1467 (127.31.32.1:36381): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3226a24d4c4146b985ad923c3ffaaee0: Network error: Client connection negotiation failed: client connection to 127.31.32.1:36381: connect: Connection refused (error 111)
I20260810 04:07:30.751336 31872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskws89Io/build/release/bin/kudu with pid 10969
I20260810 04:07:30.756788 31872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskws89Io/build/release/bin/kudu with pid 10567
2026-08-10T04:07:30Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (1964 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786334794676589-31872-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 04:07:30 dist-test-slave-8w9v krb5kdc[11164](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 04:07:30 dist-test-slave-8w9v krb5kdc[11164](info): set up 2 sockets
Aug 10 04:07:30 dist-test-slave-8w9v krb5kdc[11164](info): commencing operation
krb5kdc: starting...
W20260810 04:07:31.852074 31872 mini_kdc.cc:121] Time spent starting KDC: real 1.074s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786334794676589-31872-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786334794676589-31872-0/krb5kdc/test-user.keytab.
Aug 10 04:07:31 dist-test-slave-8w9v krb5kdc[11164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334851, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T04:07:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T04:07:31Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.31.32.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.31.32.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.31.32.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786334794676589-31872-0/krb5kdc/rangeradmin_127.31.32.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.31.32.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786334794676589-31872-0/krb5kdc/rangeradmin_127.31.32.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.31.32.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.31.32.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.31.32.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786334794676589-31872-0/krb5kdc/rangerlookup_127.31.32.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.31.32.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786334794676589-31872-0/krb5kdc/rangerlookup_127.31.32.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.31.32.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.31.32.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.31.32.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786334794676589-31872-0/krb5kdc/HTTP_127.31.32.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.31.32.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786334794676589-31872-0/krb5kdc/HTTP_127.31.32.20@KRBTEST.COM.keytab.
I20260810 04:07:32.018551 31872 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786334794676589-31872-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskws89Io/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786334794676589-31872-0/minicluster-data/postgres -l logfile start
2026-08-10 04:07:34.473 UTC [11198] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-10 04:07:34.473 UTC [11198] LOG: listening on IPv4 address "127.31.32.20", port 53651
2026-08-10 04:07:34.477 UTC [11198] LOG: listening on Unix socket "/tmp/.s.PGSQL.53651"
2026-08-10 04:07:34.512 UTC [11201] LOG: database system was shut down at 2026-08-10 04:07:33 UTC
2026-08-10 04:07:34.516 UTC [11198] LOG: database system is ready to accept connections
I20260810 04:07:35.523168 31872 mini_postgres.cc:96] Postgres bound to 53651
2026-08-10 04:07:35.530 UTC [11211] FATAL: database "slave" does not exist
127.31.32.20:53651 - accepting connections
I20260810 04:07:35.531500 31872 mini_ranger.cc:161] Starting Ranger...
I20260810 04:07:35.547613 31872 mini_ranger.cc:84] Created miniranger Postgres user
I20260810 04:07:35.989884 31872 mini_ranger.cc:87] Created ranger Postgres database
I20260810 04:07:35.989966 31872 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786334794676589-31872-0/minicluster-data/ranger-admin
2026-08-10 04:07:36,366 [I] DB FLAVOR :POSTGRES
2026-08-10 04:07:36,366 [I] --------- Verifying Ranger DB connection ---------
2026-08-10 04:07:36,366 [I] Checking connection..
2026-08-10 04:07:36,367 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskws89Io/build/release/bin/postgresql.jar:/tmp/dist-test-taskws89Io/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.32.20:53651/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-10 04:07:36,622 [I] Checking connection passed.
2026-08-10 04:07:36,622 [I] --------- Verifying version history table ---------
2026-08-10 04:07:36,622 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskws89Io/build/release/bin/postgresql.jar:/tmp/dist-test-taskws89Io/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.32.20:53651/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 04:07:36,877 [I] Table x_db_version_h does not exist in database ranger
2026-08-10 04:07:36,878 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskws89Io/build/release/bin/postgresql.jar:/tmp/dist-test-taskws89Io/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.32.20:53651/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 04:07:37,141 [I] Table x_db_version_h does not exist in database ranger
2026-08-10 04:07:37,141 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-10 04:07:37,141 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskws89Io/build/release/bin/postgresql.jar:/tmp/dist-test-taskws89Io/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.32.20:53651/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskws89Io/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-10 04:07:37.391 UTC [11279] WARNING: there is no transaction in progress
2026-08-10 04:07:37,398 [I] create_dbversion_catalog.sql file imported successfully
2026-08-10 04:07:37,399 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskws89Io/build/release/bin/postgresql.jar:/tmp/dist-test-taskws89Io/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.32.20:53651/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 04:07:37,634 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-10 04:07:37,634 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-10 04:07:37,634 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskws89Io/build/release/bin/postgresql.jar:/tmp/dist-test-taskws89Io/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.32.20:53651/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-10 04:07:37,893 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskws89Io/build/release/bin/postgresql.jar:/tmp/dist-test-taskws89Io/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.32.20:53651/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-10 04:07:38,152 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskws89Io/build/release/bin/postgresql.jar:/tmp/dist-test-taskws89Io/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.32.20:53651/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-8w9v.c.gcp-upstream.internal','N') ;"
2026-08-10 04:07:38,402 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskws89Io/build/release/bin/postgresql.jar:/tmp/dist-test-taskws89Io/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.32.20:53651/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-10 04:07:38,652 [I] Table x_portal_user does not exist in database ranger
2026-08-10 04:07:38,653 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskws89Io/build/release/bin/postgresql.jar:/tmp/dist-test-taskws89Io/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.32.20:53651/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-10 04:07:38,909 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-10 04:07:38,909 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskws89Io/build/release/bin/postgresql.jar:/tmp/dist-test-taskws89Io/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.32.20:53651/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-10 04:07:39,159 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskws89Io/build/release/bin/postgresql.jar:/tmp/dist-test-taskws89Io/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.32.20:53651/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-10 04:07:39,405 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-10 04:07:39,405 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskws89Io/build/release/bin/postgresql.jar:/tmp/dist-test-taskws89Io/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.32.20:53651/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskws89Io/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-10 04:07:40.337 UTC [11402] WARNING: there is no transaction in progress
2026-08-10 04:07:40.355 UTC [11402] WARNING: there is no transaction in progress
2026-08-10 04:07:40.370 UTC [11402] WARNING: there is no transaction in progress
2026-08-10 04:07:40.384 UTC [11402] WARNING: there is no transaction in progress
2026-08-10 04:07:40.403 UTC [11402] WARNING: there is no transaction in progress
2026-08-10 04:07:40.426 UTC [11402] WARNING: there is no transaction in progress
2026-08-10 04:07:40.495 UTC [11402] WARNING: there is no transaction in progress
2026-08-10 04:07:40.502 UTC [11402] WARNING: there is no transaction in progress
2026-08-10 04:07:40.516 UTC [11402] WARNING: there is no transaction in progress
2026-08-10 04:07:40.527 UTC [11402] WARNING: there is no transaction in progress
2026-08-10 04:07:40.537 UTC [11402] WARNING: there is no transaction in progress
2026-08-10 04:07:40.551 UTC [11402] WARNING: there is no transaction in progress
2026-08-10 04:07:40.560 UTC [11402] WARNING: there is no transaction in progress
2026-08-10 04:07:40.566 UTC [11402] WARNING: there is no transaction in progress
2026-08-10 04:07:40.573 UTC [11402] WARNING: there is no transaction in progress
2026-08-10 04:07:40.579 UTC [11402] WARNING: there is no transaction in progress
2026-08-10 04:07:41,210 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-10 04:07:41,210 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskws89Io/build/release/bin/postgresql.jar:/tmp/dist-test-taskws89Io/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.32.20:53651/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-10 04:07:41,445 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-10 04:07:41,445 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskws89Io/build/release/bin/postgresql.jar:/tmp/dist-test-taskws89Io/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.32.20:53651/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-10 04:07:41,696 [I] Table x_portal_user already exists in database 'ranger'
2026-08-10 04:07:41,697 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskws89Io/build/release/bin/postgresql.jar:/tmp/dist-test-taskws89Io/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.32.20:53651/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-10 04:07:41,961 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-10 04:07:41,962 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskws89Io/build/release/bin/postgresql.jar:/tmp/dist-test-taskws89Io/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.32.20:53651/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-10 04:07:42,220 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskws89Io/build/release/bin/postgresql.jar:/tmp/dist-test-taskws89Io/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.32.20:53651/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-10 04:07:42,466 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskws89Io/build/release/bin/postgresql.jar:/tmp/dist-test-taskws89Io/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.32.20:53651/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-8w9v.c.gcp-upstream.internal';"
2026-08-10 04:07:42,709 [I] CORE_DB_SCHEMA import status has been updated
2026-08-10 04:07:42,710 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskws89Io/build/release/bin/postgresql.jar:/tmp/dist-test-taskws89Io/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.32.20:53651/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-10 04:07:42,963 [I] DB_PATCHES have already been applied
I20260810 04:07:42.969244 31872 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786334794676589-31872-0/minicluster-data/ranger-admin:/tmp/dist-test-taskws89Io/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskws89Io/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskws89Io/build/release/bin/postgresql.jar:/tmp/dist-test-taskws89Io/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260810 04:07:42.969312 31872 mini_ranger.cc:193] Using host: 127.31.32.20
I20260810 04:07:42.972918 31872 mini_ranger.cc:227] Ranger admin URL: http://127.31.32.20:52053
Aug 10, 2026 4:07:43 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 10, 2026 4:07:43 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 10, 2026 4:07:43 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786334794676589-31872-0/minicluster-data/ranger-admin/ews/webapp
Aug 10, 2026 4:07:43 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786334794676589-31872-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 10, 2026 4:07:43 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786334794676589-31872-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 10, 2026 4:07:43 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786334794676589-31872-0/minicluster-data/ranger-admin/ews/webapp].
Aug 10, 2026 4:07:43 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260810 04:07:44.146158 31872 mini_ranger.cc:160] Time spent starting Ranger: real 8.615s user 0.000s sys 0.010s
Aug 10, 2026 4:08:04 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 10, 2026 4:08:04 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.MetricsREST
Aug 10, 2026 4:08:04 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 10, 2026 4:08:04 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 10, 2026 4:08:04 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 10, 2026 4:08:04 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 10, 2026 4:08:04 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 10, 2026 4:08:04 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 10, 2026 4:08:04 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 10, 2026 4:08:04 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 10, 2026 4:08:04 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 10, 2026 4:08:04 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 10, 2026 4:08:04 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 10, 2026 4:08:04 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 10, 2026 4:08:04 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 10, 2026 4:08:04 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 10, 2026 4:08:04 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 10, 2026 4:08:05 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260810 04:08:06.326841 31872 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.31.32.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.31.32.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.31.32.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786334794676589-31872-0/krb5kdc/rangerkms_127.31.32.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.31.32.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786334794676589-31872-0/krb5kdc/rangerkms_127.31.32.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.31.32.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.31.32.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.31.32.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786334794676589-31872-0/krb5kdc/HTTP_127.31.32.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.31.32.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786334794676589-31872-0/krb5kdc/HTTP_127.31.32.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 10 04:08:06 dist-test-slave-8w9v krb5kdc[11164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334886, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260810 04:08:06.422803 31872 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260810 04:08:06.432763 31872 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260810 04:08:07.246052 31872 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260810 04:08:07.246155 31872 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786334794676589-31872-0/minicluster-data/ranger-kms
I20260810 04:08:07.246179 31872 mini_ranger_kms.cc:227] Using postgres at 127.31.32.20:53651
2026-08-10 04:08:07,604 [I] DB FLAVOR :POSTGRES
2026-08-10 04:08:07,605 [I] --------- Verifying Ranger DB connection ---------
2026-08-10 04:08:07,605 [I] Checking connection
2026-08-10 04:08:07,605 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskws89Io/build/release/bin/postgresql.jar:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786334794676589-31872-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.32.20:53651/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-10 04:08:07,854 [I] connection success
2026-08-10 04:08:07,855 [I] --------- Verifying Ranger DB tables ---------
2026-08-10 04:08:07,855 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-10 04:08:07,855 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskws89Io/build/release/bin/postgresql.jar:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786334794676589-31872-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.32.20:53651/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-10 04:08:08,099 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-10 04:08:08,099 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-10 04:08:08,099 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-10 04:08:08,100 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskws89Io/build/release/bin/postgresql.jar:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786334794676589-31872-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.32.20:53651/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786334794676589-31872-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-10 04:08:08,374 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260810 04:08:08.863402 31872 mini_ranger_kms.cc:310] Created kms service
I20260810 04:08:09.035060 31872 mini_ranger_kms.cc:326] Created kudu user
I20260810 04:08:09.081103 31872 mini_ranger_kms.cc:343] Created rangerkms user
Aug 10 04:08:09 dist-test-slave-8w9v krb5kdc[11164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334886, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.31.32.20@KRBTEST.COM
I20260810 04:08:09.449971 31872 mini_ranger_kms.cc:384] Added ranger policy
I20260810 04:08:09.450086 31872 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786334794676589-31872-0/minicluster-data/ranger-kms:/tmp/dist-test-taskws89Io/build/release/bin/postgresql.jar:/tmp/dist-test-taskws89Io/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskws89Io/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260810 04:08:09.450111 31872 mini_ranger_kms.cc:240] Using host: 127.31.32.20
I20260810 04:08:09.454293 31872 mini_ranger_kms.cc:276] Ranger KMS PID: 11608
I20260810 04:08:09.454425 31872 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.31.32.20:43087
Aug 10, 2026 4:08:10 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 10, 2026 4:08:10 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 10, 2026 4:08:10 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786334794676589-31872-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 10, 2026 4:08:10 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786334794676589-31872-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 10, 2026 4:08:11 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786334794676589-31872-0/minicluster-data/ranger-kms/ews/webapp].
Aug 10, 2026 4:08:11 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260810 04:08:11.865199 31872 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.442s user 0.004s sys 0.250s
I20260810 04:08:11.865329 31872 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260810 04:08:11.865411 31872 mini_ranger_kms.cc:401] 127.31.32.20:43087/kms/v1/keys
Aug 10 04:08:12 dist-test-slave-8w9v krb5kdc[11164](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786334892, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.31.32.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Aug 10 04:08:12 dist-test-slave-8w9v krb5kdc[11164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334892, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.31.32.20@KRBTEST.COM for HTTP/127.31.32.20@KRBTEST.COM
Aug 10 04:08:12 dist-test-slave-8w9v krb5kdc[11164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334892, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.31.32.20@KRBTEST.COM for HTTP/127.31.32.20@KRBTEST.COM
Aug 10 04:08:15 dist-test-slave-8w9v krb5kdc[11164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334895, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.31.32.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.32.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.32.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.62.keytab.
Entry for principal kudu/127.31.32.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.62.keytab.
I20260810 04:08:15.141006 31872 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskws89Io/build/release/bin/kudu
/tmp/dist-test-taskws89Io/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786334794676589-31872-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786334794676589-31872-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786334794676589-31872-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786334794676589-31872-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.32.62:34735
--webserver_interface=127.31.32.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.62.keytab
--principal=kudu/127.31.32.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.32.20:42689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.32.62:34735
--ranger_config_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786334794676589-31872-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.32.20:43087/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786334794676589-31872-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786334794676589-31872-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786334794676589-31872-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:08:15.237668 11666 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:08:15.237876 11666 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:08:15.237897 11666 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:08:15.239046 11666 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:08:15.239089 11666 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:08:15.239102 11666 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:08:15.239114 11666 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:08:15.239126 11666 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:08:15.240813 11666 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.32.20:42689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786334794676589-31872-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786334794676589-31872-0/minicluster-data/master-0/wal
--ranger_kms_url=127.31.32.20:43087/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.31.32.62:34735
--ranger_config_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786334794676589-31872-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.62.keytab
--principal=kudu/127.31.32.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.32.62:34735
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786334794676589-31872-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.32.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11666
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786334794676589-31872-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:08:15.241065 11666 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:08:15.241320 11666 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:08:15.247458 11673 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:08:15.247581 11675 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:08:15.247520 11666 server_base.cc:1034] running on GCE node
W20260810 04:08:15.247464 11672 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:08:15.248001 11666 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:08:15.248376 11666 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:08:15.249569 11666 hybrid_clock.cc:648] HybridClock initialized: now 1786334895249545 us; error 42 us; skew 500 ppm
Aug 10 04:08:15 dist-test-slave-8w9v krb5kdc[11164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334895, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:08:15.252652 11666 init.cc:377] Logged in from keytab as kudu/127.31.32.62@KRBTEST.COM (short username kudu)
I20260810 04:08:15.253136 11666 webserver.cc:460] Webserver started at http://127.31.32.62:46681/ using document root <none> and password file <none>
I20260810 04:08:15.253389 11666 fs_manager.cc:359] Metadata directory not provided
I20260810 04:08:15.253453 11666 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:08:15.253611 11666 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 04:08:15 dist-test-slave-8w9v krb5kdc[11164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334895, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.62@KRBTEST.COM for HTTP/127.31.32.20@KRBTEST.COM
I20260810 04:08:15.315801 11666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786334794676589-31872-0/minicluster-data/master-0/data/instance:
uuid: "f1e5112d255148e18c83e49f537a5be6"
format_stamp: "Formatted at 2026-08-10 04:08:15 on dist-test-slave-8w9v"
server_key: "c608aafc89c536ac3c51e8d515a93f3b"
server_key_iv: "b27a52173f39ee4552cc3b2eed31e9f6"
server_key_version: "kuduclusterkey@0"
I20260810 04:08:15.316357 11666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786334794676589-31872-0/minicluster-data/master-0/wal/instance:
uuid: "f1e5112d255148e18c83e49f537a5be6"
format_stamp: "Formatted at 2026-08-10 04:08:15 on dist-test-slave-8w9v"
server_key: "c608aafc89c536ac3c51e8d515a93f3b"
server_key_iv: "b27a52173f39ee4552cc3b2eed31e9f6"
server_key_version: "kuduclusterkey@0"
I20260810 04:08:15.318058 11666 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260810 04:08:15.330700 11682 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:08:15.330958 11666 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260810 04:08:15.331071 11666 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786334794676589-31872-0/minicluster-data/master-0/data,/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786334794676589-31872-0/minicluster-data/master-0/wal
uuid: "f1e5112d255148e18c83e49f537a5be6"
format_stamp: "Formatted at 2026-08-10 04:08:15 on dist-test-slave-8w9v"
server_key: "c608aafc89c536ac3c51e8d515a93f3b"
server_key_iv: "b27a52173f39ee4552cc3b2eed31e9f6"
server_key_version: "kuduclusterkey@0"
I20260810 04:08:15.331190 11666 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786334794676589-31872-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786334794676589-31872-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786334794676589-31872-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:08:15.359808 11666 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:08:15.361908 11666 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:08:15.362071 11666 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:08:15.367790 11666 rpc_server.cc:307] RPC server started. Bound to: 127.31.32.62:34735
I20260810 04:08:15.367887 11741 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.32.62:34735 every 8 connection(s)
I20260810 04:08:15.368331 11666 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786334794676589-31872-0/minicluster-data/master-0/data/info.pb
I20260810 04:08:15.369108 11742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:08:15.371848 11742 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1e5112d255148e18c83e49f537a5be6: Bootstrap starting.
I20260810 04:08:15.372542 11742 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f1e5112d255148e18c83e49f537a5be6: Neither blocks nor log segments found. Creating new log.
I20260810 04:08:15.372876 11742 log.cc:824] T 00000000000000000000000000000000 P f1e5112d255148e18c83e49f537a5be6: Log is configured to *not* fsync() on all Append() calls
I20260810 04:08:15.373693 11742 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1e5112d255148e18c83e49f537a5be6: No bootstrap required, opened a new log
I20260810 04:08:15.375939 11742 raft_consensus.cc:348] T 00000000000000000000000000000000 P f1e5112d255148e18c83e49f537a5be6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1e5112d255148e18c83e49f537a5be6" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 34735 } }
I20260810 04:08:15.376135 11742 raft_consensus.cc:374] T 00000000000000000000000000000000 P f1e5112d255148e18c83e49f537a5be6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:08:15.376217 11742 raft_consensus.cc:729] T 00000000000000000000000000000000 P f1e5112d255148e18c83e49f537a5be6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e5112d255148e18c83e49f537a5be6, State: Initialized, Role: FOLLOWER
I20260810 04:08:15.376408 11742 consensus_queue.cc:260] T 00000000000000000000000000000000 P f1e5112d255148e18c83e49f537a5be6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1e5112d255148e18c83e49f537a5be6" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 34735 } }
I20260810 04:08:15.376521 11742 raft_consensus.cc:388] T 00000000000000000000000000000000 P f1e5112d255148e18c83e49f537a5be6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 04:08:15.376570 11742 raft_consensus.cc:482] T 00000000000000000000000000000000 P f1e5112d255148e18c83e49f537a5be6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 04:08:15.376626 11742 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f1e5112d255148e18c83e49f537a5be6 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:08:15.377246 11742 raft_consensus.cc:504] T 00000000000000000000000000000000 P f1e5112d255148e18c83e49f537a5be6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1e5112d255148e18c83e49f537a5be6" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 34735 } }
I20260810 04:08:15.377384 11742 leader_election.cc:302] T 00000000000000000000000000000000 P f1e5112d255148e18c83e49f537a5be6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f1e5112d255148e18c83e49f537a5be6; no voters:
I20260810 04:08:15.377600 11742 leader_election.cc:288] T 00000000000000000000000000000000 P f1e5112d255148e18c83e49f537a5be6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 04:08:15.377628 11745 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f1e5112d255148e18c83e49f537a5be6 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:08:15.377766 11745 raft_consensus.cc:686] T 00000000000000000000000000000000 P f1e5112d255148e18c83e49f537a5be6 [term 1 LEADER]: Becoming Leader. State: Replica: f1e5112d255148e18c83e49f537a5be6, State: Running, Role: LEADER
I20260810 04:08:15.377941 11742 sys_catalog.cc:564] T 00000000000000000000000000000000 P f1e5112d255148e18c83e49f537a5be6 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:08:15.377919 11745 consensus_queue.cc:237] T 00000000000000000000000000000000 P f1e5112d255148e18c83e49f537a5be6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1e5112d255148e18c83e49f537a5be6" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 34735 } }
I20260810 04:08:15.378088 31872 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskws89Io/build/release/bin/kudu as pid 11666
I20260810 04:08:15.378180 31872 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786334794676589-31872-0/minicluster-data/master-0/wal/instance
I20260810 04:08:15.378393 31872 external_mini_cluster.cc:1428] Setting key ec2280d6a3ef1c86167bc2ff3f831511
I20260810 04:08:15.378475 11747 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1e5112d255148e18c83e49f537a5be6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f1e5112d255148e18c83e49f537a5be6. Latest consensus state: current_term: 1 leader_uuid: "f1e5112d255148e18c83e49f537a5be6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1e5112d255148e18c83e49f537a5be6" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 34735 } } }
I20260810 04:08:15.379071 11747 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1e5112d255148e18c83e49f537a5be6 [sys.catalog]: This master's current role is: LEADER
I20260810 04:08:15.378654 11746 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1e5112d255148e18c83e49f537a5be6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f1e5112d255148e18c83e49f537a5be6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1e5112d255148e18c83e49f537a5be6" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 34735 } } }
I20260810 04:08:15.379204 11746 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1e5112d255148e18c83e49f537a5be6 [sys.catalog]: This master's current role is: LEADER
I20260810 04:08:15.379518 11756 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 04:08:15.380054 11756 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 04:08:15.380074 11742 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786334794676589-31872-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260810 04:08:15.386664 11756 catalog_manager.cc:1347] Generated new cluster ID: 541eb5af84b6439d9caffe245b283a81
I20260810 04:08:15.386837 11756 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 10 04:08:15 dist-test-slave-8w9v krb5kdc[11164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334895, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.32.62@KRBTEST.COM
I20260810 04:08:15.397329 11755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:08:15.378888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55254 (local address 127.31.32.62:34735)
0810 04:08:15.380461 (+ 1573us) server_negotiation.cc:206] Beginning negotiation
0810 04:08:15.380469 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 04:08:15.380495 (+ 26us) server_negotiation.cc:407] Connection header received
0810 04:08:15.380698 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:08:15.380707 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:08:15.380781 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:08:15.380924 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:08:15.384720 (+ 3796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:08:15.387055 (+ 2335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:08:15.387917 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:08:15.388735 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:08:15.388864 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:08:15.391593 (+ 2729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:08:15.391597 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:08:15.391598 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:08:15.391610 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:08:15.395268 (+ 3658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:08:15.395665 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:08:15.395668 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:08:15.395668 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:08:15.395793 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:08:15.396091 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:08:15.396092 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:08:15.396092 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:08:15.396508 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:08:15.396698 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0810 04:08:15.396788 (+ 90us) server_negotiation.cc:299] Negotiation successful
0810 04:08:15.396928 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1530,"thread_start_us":196,"threads_started":1}
I20260810 04:08:15.403440 11756 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 04:08:15.404207 11756 catalog_manager.cc:1504] Loading token signing keys...
I20260810 04:08:15.406510 11756 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f1e5112d255148e18c83e49f537a5be6: Generated new TSK 0
I20260810 04:08:15.406725 11756 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.32.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.32.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.32.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.1.keytab.
Entry for principal kudu/127.31.32.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.1.keytab.
I20260810 04:08:16.154950 31872 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskws89Io/build/release/bin/kudu
/tmp/dist-test-taskws89Io/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786334794676589-31872-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786334794676589-31872-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786334794676589-31872-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786334794676589-31872-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.32.20:43087/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.32.1:0
--local_ip_for_outbound_sockets=127.31.32.1
--webserver_interface=127.31.32.1
--webserver_port=0
--tserver_master_addrs=127.31.32.62:34735
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.1.keytab
--principal=kudu/127.31.32.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.32.20:42689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786334794676589-31872-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786334794676589-31872-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786334794676589-31872-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:08:16.246513 11790 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:08:16.246737 11790 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:08:16.246783 11790 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:08:16.247941 11790 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:08:16.248003 11790 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:08:16.248049 11790 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.32.1
I20260810 04:08:16.249536 11790 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.32.20:42689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786334794676589-31872-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786334794676589-31872-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.31.32.20:43087/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.1.keytab
--principal=kudu/127.31.32.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.32.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786334794676589-31872-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.32.1
--webserver_port=0
--tserver_master_addrs=127.31.32.62:34735
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11790
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.32.1
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786334794676589-31872-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:08:16.249835 11790 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:08:16.250094 11790 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:08:16.255467 11797 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:08:16.255591 11798 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:08:16.255452 11800 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:08:16.255860 11790 server_base.cc:1034] running on GCE node
I20260810 04:08:16.256175 11790 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:08:16.256498 11790 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:08:16.257702 11790 hybrid_clock.cc:648] HybridClock initialized: now 1786334896257678 us; error 40 us; skew 500 ppm
Aug 10 04:08:16 dist-test-slave-8w9v krb5kdc[11164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334896, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:08:16.260629 11790 init.cc:377] Logged in from keytab as kudu/127.31.32.1@KRBTEST.COM (short username kudu)
I20260810 04:08:16.261174 11790 webserver.cc:460] Webserver started at http://127.31.32.1:38557/ using document root <none> and password file <none>
I20260810 04:08:16.261451 11790 fs_manager.cc:359] Metadata directory not provided
I20260810 04:08:16.261503 11790 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:08:16.261649 11790 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 04:08:16 dist-test-slave-8w9v krb5kdc[11164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334896, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.1@KRBTEST.COM for HTTP/127.31.32.20@KRBTEST.COM
I20260810 04:08:16.274945 11790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786334794676589-31872-0/minicluster-data/ts-0/data/instance:
uuid: "ec06548e7ccc422e893f208652e3b939"
format_stamp: "Formatted at 2026-08-10 04:08:16 on dist-test-slave-8w9v"
server_key: "d57b2236c5f2042c6af161d7069f78b3"
server_key_iv: "f27ca8e5bad05d2b56343fed56a06b0d"
server_key_version: "kuduclusterkey@0"
I20260810 04:08:16.275409 11790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786334794676589-31872-0/minicluster-data/ts-0/wal/instance:
uuid: "ec06548e7ccc422e893f208652e3b939"
format_stamp: "Formatted at 2026-08-10 04:08:16 on dist-test-slave-8w9v"
server_key: "d57b2236c5f2042c6af161d7069f78b3"
server_key_iv: "f27ca8e5bad05d2b56343fed56a06b0d"
server_key_version: "kuduclusterkey@0"
I20260810 04:08:16.277187 11790 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260810 04:08:16.295182 11808 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:08:16.295449 11790 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20260810 04:08:16.295567 11790 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786334794676589-31872-0/minicluster-data/ts-0/data,/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786334794676589-31872-0/minicluster-data/ts-0/wal
uuid: "ec06548e7ccc422e893f208652e3b939"
format_stamp: "Formatted at 2026-08-10 04:08:16 on dist-test-slave-8w9v"
server_key: "d57b2236c5f2042c6af161d7069f78b3"
server_key_iv: "f27ca8e5bad05d2b56343fed56a06b0d"
server_key_version: "kuduclusterkey@0"
I20260810 04:08:16.295701 11790 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786334794676589-31872-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786334794676589-31872-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786334794676589-31872-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:08:16.313329 11790 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:08:16.315969 11790 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:08:16.316217 11790 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:08:16.316538 11790 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:08:16.316941 11790 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:08:16.316984 11790 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:08:16.317015 11790 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:08:16.317036 11790 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:08:16.325112 11790 rpc_server.cc:307] RPC server started. Bound to: 127.31.32.1:34889
I20260810 04:08:16.325305 11932 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.32.1:34889 every 8 connection(s)
I20260810 04:08:16.325575 11790 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786334794676589-31872-0/minicluster-data/ts-0/data/info.pb
I20260810 04:08:16.331144 31872 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskws89Io/build/release/bin/kudu as pid 11790
I20260810 04:08:16.331239 31872 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786334794676589-31872-0/minicluster-data/ts-0/wal/instance
I20260810 04:08:16.331444 31872 external_mini_cluster.cc:1428] Setting key ff51081cefd82e0640db4bfd2cb55299
Aug 10 04:08:16 dist-test-slave-8w9v krb5kdc[11164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334896, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.1@KRBTEST.COM for kudu/127.31.32.62@KRBTEST.COM
I20260810 04:08:16.339097 11936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:08:16.326487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:34735 (local address 127.31.32.1:55489)
0810 04:08:16.328426 (+ 1939us) negotiation.cc:107] Waiting for socket to connect
0810 04:08:16.328462 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0810 04:08:16.329016 (+ 554us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:08:16.329305 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:08:16.329310 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:08:16.329510 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:08:16.329711 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:08:16.329715 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:08:16.330481 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:08:16.330482 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:08:16.331934 (+ 1452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:08:16.331937 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:08:16.332684 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:08:16.332686 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:08:16.332858 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:08:16.333454 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:08:16.333465 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:08:16.335071 (+ 1606us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:08:16.337298 (+ 2227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:08:16.337304 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:08:16.337306 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:08:16.337645 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:08:16.337889 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:08:16.337890 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:08:16.337891 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:08:16.338047 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:08:16.338370 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:08:16.338377 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:08:16.338578 (+ 201us) client_negotiation.cc:769] Sending connection context
0810 04:08:16.338693 (+ 115us) client_negotiation.cc:240] Negotiation successful
0810 04:08:16.338866 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1903,"thread_start_us":97,"threads_started":1}
I20260810 04:08:16.339540 11935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:08:16.326504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.1:55489 (local address 127.31.32.62:34735)
0810 04:08:16.328373 (+ 1869us) server_negotiation.cc:206] Beginning negotiation
0810 04:08:16.328375 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:08:16.328558 (+ 183us) server_negotiation.cc:407] Connection header received
0810 04:08:16.329134 (+ 576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:08:16.329135 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:08:16.329150 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:08:16.329229 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:08:16.329820 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:08:16.330378 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:08:16.332028 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:08:16.332551 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:08:16.332630 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:08:16.335222 (+ 2592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:08:16.335224 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:08:16.335224 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:08:16.335233 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:08:16.337189 (+ 1956us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:08:16.337735 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:08:16.337736 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:08:16.337736 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:08:16.337800 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:08:16.338129 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:08:16.338130 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:08:16.338131 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:08:16.338273 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:08:16.338340 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0810 04:08:16.339366 (+ 1026us) server_negotiation.cc:299] Negotiation successful
0810 04:08:16.339449 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1836,"thread_start_us":86,"threads_started":1}
I20260810 04:08:16.339879 11933 heartbeater.cc:347] Connected to a master server at 127.31.32.62:34735
I20260810 04:08:16.339965 11933 heartbeater.cc:464] Registering TS with master...
I20260810 04:08:16.340181 11933 heartbeater.cc:511] Master 127.31.32.62:34735 requested a full tablet report, sending...
I20260810 04:08:16.340685 11702 ts_manager.cc:194] Registered new tserver with Master: ec06548e7ccc422e893f208652e3b939 (127.31.32.1:34889)
I20260810 04:08:16.341512 11702 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.32.1@KRBTEST.COM'} at 127.31.32.1:55489
WARNING: no policy specified for kudu/127.31.32.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.32.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.32.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.2.keytab.
Entry for principal kudu/127.31.32.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.2.keytab.
I20260810 04:08:16.380656 31872 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskws89Io/build/release/bin/kudu
/tmp/dist-test-taskws89Io/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786334794676589-31872-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786334794676589-31872-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786334794676589-31872-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786334794676589-31872-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.32.20:43087/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.32.2:0
--local_ip_for_outbound_sockets=127.31.32.2
--webserver_interface=127.31.32.2
--webserver_port=0
--tserver_master_addrs=127.31.32.62:34735
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.2.keytab
--principal=kudu/127.31.32.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.32.20:42689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786334794676589-31872-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786334794676589-31872-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786334794676589-31872-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:08:16.503031 11944 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:08:16.503357 11944 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:08:16.503448 11944 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:08:16.505103 11944 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:08:16.505213 11944 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:08:16.505290 11944 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.32.2
I20260810 04:08:16.507413 11944 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.32.20:42689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786334794676589-31872-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786334794676589-31872-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.31.32.20:43087/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.2.keytab
--principal=kudu/127.31.32.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.32.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786334794676589-31872-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.32.2
--webserver_port=0
--tserver_master_addrs=127.31.32.62:34735
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11944
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.32.2
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786334794676589-31872-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:08:16.507867 11944 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:08:16.508208 11944 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:08:16.521540 11952 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:08:16.521541 11951 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:08:16.522315 11955 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:08:16.522364 11944 server_base.cc:1034] running on GCE node
I20260810 04:08:16.522714 11944 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:08:16.523031 11944 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:08:16.526317 11944 hybrid_clock.cc:648] HybridClock initialized: now 1786334896526284 us; error 49 us; skew 500 ppm
Aug 10 04:08:16 dist-test-slave-8w9v krb5kdc[11164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334896, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:08:16.530012 11944 init.cc:377] Logged in from keytab as kudu/127.31.32.2@KRBTEST.COM (short username kudu)
I20260810 04:08:16.530584 11944 webserver.cc:460] Webserver started at http://127.31.32.2:40971/ using document root <none> and password file <none>
I20260810 04:08:16.530853 11944 fs_manager.cc:359] Metadata directory not provided
I20260810 04:08:16.530918 11944 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:08:16.531080 11944 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 04:08:16 dist-test-slave-8w9v krb5kdc[11164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334896, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.2@KRBTEST.COM for HTTP/127.31.32.20@KRBTEST.COM
I20260810 04:08:16.549933 11944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786334794676589-31872-0/minicluster-data/ts-1/data/instance:
uuid: "574bf888816c47f69bc82232cfc36ef7"
format_stamp: "Formatted at 2026-08-10 04:08:16 on dist-test-slave-8w9v"
server_key: "260a4c86e06ce5ae6fb9d9c29b252b6b"
server_key_iv: "da23714509963c84de1ad8e908a32290"
server_key_version: "kuduclusterkey@0"
I20260810 04:08:16.550551 11944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786334794676589-31872-0/minicluster-data/ts-1/wal/instance:
uuid: "574bf888816c47f69bc82232cfc36ef7"
format_stamp: "Formatted at 2026-08-10 04:08:16 on dist-test-slave-8w9v"
server_key: "260a4c86e06ce5ae6fb9d9c29b252b6b"
server_key_iv: "da23714509963c84de1ad8e908a32290"
server_key_version: "kuduclusterkey@0"
I20260810 04:08:16.554647 11944 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s
I20260810 04:08:16.572069 11962 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:08:16.572779 11944 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 04:08:16.572959 11944 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786334794676589-31872-0/minicluster-data/ts-1/data,/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786334794676589-31872-0/minicluster-data/ts-1/wal
uuid: "574bf888816c47f69bc82232cfc36ef7"
format_stamp: "Formatted at 2026-08-10 04:08:16 on dist-test-slave-8w9v"
server_key: "260a4c86e06ce5ae6fb9d9c29b252b6b"
server_key_iv: "da23714509963c84de1ad8e908a32290"
server_key_version: "kuduclusterkey@0"
I20260810 04:08:16.573132 11944 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786334794676589-31872-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786334794676589-31872-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786334794676589-31872-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
Aug 10 04:08:16 dist-test-slave-8w9v krb5kdc[11164](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786334896, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:08:16.594753 11944 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:08:16.597343 11944 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:08:16.597509 11944 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:08:16.597780 11944 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:08:16.598188 11944 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:08:16.598232 11944 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:08:16.598281 11944 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:08:16.598306 11944 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:08:16.615268 11944 rpc_server.cc:307] RPC server started. Bound to: 127.31.32.2:41581
I20260810 04:08:16.615407 12085 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.32.2:41581 every 8 connection(s)
I20260810 04:08:16.615813 11944 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786334794676589-31872-0/minicluster-data/ts-1/data/info.pb
I20260810 04:08:16.620209 31872 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskws89Io/build/release/bin/kudu as pid 11944
I20260810 04:08:16.620306 31872 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786334794676589-31872-0/minicluster-data/ts-1/wal/instance
I20260810 04:08:16.620481 31872 external_mini_cluster.cc:1428] Setting key 0c2066acca46cf844593f3e8b10f0141
Aug 10 04:08:16 dist-test-slave-8w9v krb5kdc[11164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334896, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.2@KRBTEST.COM for kudu/127.31.32.62@KRBTEST.COM
I20260810 04:08:16.628181 12088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:08:16.616775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:34735 (local address 127.31.32.2:42169)
0810 04:08:16.616962 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0810 04:08:16.617028 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0810 04:08:16.617794 (+ 766us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:08:16.618099 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:08:16.618104 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:08:16.618398 (+ 294us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:08:16.618711 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:08:16.618719 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:08:16.619695 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:08:16.619697 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:08:16.621179 (+ 1482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:08:16.621183 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:08:16.621846 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:08:16.621847 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:08:16.622021 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:08:16.622687 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:08:16.622698 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:08:16.624468 (+ 1770us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:08:16.626446 (+ 1978us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:08:16.626454 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:08:16.626457 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:08:16.626752 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:08:16.626997 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:08:16.626998 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:08:16.626998 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:08:16.627142 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:08:16.627472 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:08:16.627477 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:08:16.627642 (+ 165us) client_negotiation.cc:769] Sending connection context
0810 04:08:16.627744 (+ 102us) client_negotiation.cc:240] Negotiation successful
0810 04:08:16.627901 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":94,"threads_started":1}
I20260810 04:08:16.628989 11935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:08:16.616810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.2:42169 (local address 127.31.32.62:34735)
0810 04:08:16.616861 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0810 04:08:16.616863 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:08:16.617178 (+ 315us) server_negotiation.cc:407] Connection header received
0810 04:08:16.617917 (+ 739us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:08:16.617919 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:08:16.617932 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:08:16.618005 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:08:16.618825 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:08:16.619580 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:08:16.621314 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:08:16.621661 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:08:16.621706 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:08:16.624586 (+ 2880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:08:16.624588 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:08:16.624588 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:08:16.624595 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:08:16.626346 (+ 1751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:08:16.626870 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:08:16.626871 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:08:16.626871 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:08:16.626930 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:08:16.627238 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:08:16.627239 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:08:16.627239 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:08:16.627391 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:08:16.627449 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0810 04:08:16.628786 (+ 1337us) server_negotiation.cc:299] Negotiation successful
0810 04:08:16.628890 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260810 04:08:16.630435 12086 heartbeater.cc:347] Connected to a master server at 127.31.32.62:34735
I20260810 04:08:16.630543 12086 heartbeater.cc:464] Registering TS with master...
I20260810 04:08:16.630756 12086 heartbeater.cc:511] Master 127.31.32.62:34735 requested a full tablet report, sending...
I20260810 04:08:16.631139 11702 ts_manager.cc:194] Registered new tserver with Master: 574bf888816c47f69bc82232cfc36ef7 (127.31.32.2:41581)
I20260810 04:08:16.631721 11702 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.32.2@KRBTEST.COM'} at 127.31.32.2:42169
WARNING: no policy specified for kudu/127.31.32.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.32.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.32.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.3.keytab.
Entry for principal kudu/127.31.32.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.3.keytab.
I20260810 04:08:16.663854 31872 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskws89Io/build/release/bin/kudu
/tmp/dist-test-taskws89Io/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786334794676589-31872-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786334794676589-31872-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786334794676589-31872-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786334794676589-31872-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.32.20:43087/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.32.3:0
--local_ip_for_outbound_sockets=127.31.32.3
--webserver_interface=127.31.32.3
--webserver_port=0
--tserver_master_addrs=127.31.32.62:34735
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.3.keytab
--principal=kudu/127.31.32.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.32.20:42689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786334794676589-31872-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786334794676589-31872-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786334794676589-31872-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:08:16.781826 12092 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:08:16.782166 12092 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:08:16.782250 12092 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:08:16.783950 12092 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:08:16.784046 12092 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:08:16.784155 12092 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.32.3
I20260810 04:08:16.786389 12092 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.32.20:42689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786334794676589-31872-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786334794676589-31872-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.31.32.20:43087/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.3.keytab
--principal=kudu/127.31.32.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.32.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786334794676589-31872-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.32.3
--webserver_port=0
--tserver_master_addrs=127.31.32.62:34735
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12092
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.32.3
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786334794676589-31872-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:08:16.786875 12092 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:08:16.787204 12092 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:08:16.793917 12100 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:08:16.794353 12102 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:08:16.795117 12098 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:08:16.795656 12092 server_base.cc:1034] running on GCE node
I20260810 04:08:16.795959 12092 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:08:16.796252 12092 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:08:16.799471 12092 hybrid_clock.cc:648] HybridClock initialized: now 1786334896799446 us; error 38 us; skew 500 ppm
Aug 10 04:08:16 dist-test-slave-8w9v krb5kdc[11164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334896, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:08:16.802641 12092 init.cc:377] Logged in from keytab as kudu/127.31.32.3@KRBTEST.COM (short username kudu)
I20260810 04:08:16.803114 12092 webserver.cc:460] Webserver started at http://127.31.32.3:36517/ using document root <none> and password file <none>
I20260810 04:08:16.803380 12092 fs_manager.cc:359] Metadata directory not provided
I20260810 04:08:16.803439 12092 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:08:16.803562 12092 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 04:08:16 dist-test-slave-8w9v krb5kdc[11164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334896, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.3@KRBTEST.COM for HTTP/127.31.32.20@KRBTEST.COM
I20260810 04:08:16.819774 12092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786334794676589-31872-0/minicluster-data/ts-2/data/instance:
uuid: "9fccc96ef747484bb7b8ddfbc7c04c75"
format_stamp: "Formatted at 2026-08-10 04:08:16 on dist-test-slave-8w9v"
server_key: "a83ad2e2e118bc05bb3e58c8d221bb06"
server_key_iv: "e6093494cf74216e65588cc2f3c930eb"
server_key_version: "kuduclusterkey@0"
I20260810 04:08:16.820227 12092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786334794676589-31872-0/minicluster-data/ts-2/wal/instance:
uuid: "9fccc96ef747484bb7b8ddfbc7c04c75"
format_stamp: "Formatted at 2026-08-10 04:08:16 on dist-test-slave-8w9v"
server_key: "a83ad2e2e118bc05bb3e58c8d221bb06"
server_key_iv: "e6093494cf74216e65588cc2f3c930eb"
server_key_version: "kuduclusterkey@0"
I20260810 04:08:16.822908 12092 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260810 04:08:16.845513 12109 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:08:16.845685 12092 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260810 04:08:16.845779 12092 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786334794676589-31872-0/minicluster-data/ts-2/data,/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786334794676589-31872-0/minicluster-data/ts-2/wal
uuid: "9fccc96ef747484bb7b8ddfbc7c04c75"
format_stamp: "Formatted at 2026-08-10 04:08:16 on dist-test-slave-8w9v"
server_key: "a83ad2e2e118bc05bb3e58c8d221bb06"
server_key_iv: "e6093494cf74216e65588cc2f3c930eb"
server_key_version: "kuduclusterkey@0"
I20260810 04:08:16.845883 12092 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786334794676589-31872-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786334794676589-31872-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786334794676589-31872-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:08:16.879578 12092 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:08:16.882148 12092 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:08:16.882294 12092 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:08:16.882681 12092 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:08:16.883123 12092 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:08:16.883183 12092 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:08:16.883217 12092 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:08:16.883235 12092 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:08:16.891135 12092 rpc_server.cc:307] RPC server started. Bound to: 127.31.32.3:43187
I20260810 04:08:16.891595 12092 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786334794676589-31872-0/minicluster-data/ts-2/data/info.pb
I20260810 04:08:16.894016 12239 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.32.3:43187 every 8 connection(s)
I20260810 04:08:16.900900 31872 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskws89Io/build/release/bin/kudu as pid 12092
I20260810 04:08:16.901014 31872 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786334794676589-31872-0/minicluster-data/ts-2/wal/instance
I20260810 04:08:16.901209 31872 external_mini_cluster.cc:1428] Setting key 8210f8c8cb32962f911472e2f80b912c
Aug 10 04:08:16 dist-test-slave-8w9v krb5kdc[11164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334896, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.3@KRBTEST.COM for kudu/127.31.32.62@KRBTEST.COM
I20260810 04:08:16.905435 11935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:08:16.894985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.3:39951 (local address 127.31.32.62:34735)
0810 04:08:16.895032 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0810 04:08:16.895034 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:08:16.895423 (+ 389us) server_negotiation.cc:407] Connection header received
0810 04:08:16.896093 (+ 670us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:08:16.896095 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:08:16.896107 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:08:16.896155 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:08:16.896982 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:08:16.897793 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:08:16.899255 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:08:16.899550 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:08:16.899594 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:08:16.902144 (+ 2550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:08:16.902145 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:08:16.902146 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:08:16.902153 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:08:16.903851 (+ 1698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:08:16.904342 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:08:16.904343 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:08:16.904344 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:08:16.904400 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:08:16.904676 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:08:16.904677 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:08:16.904677 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:08:16.904806 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:08:16.904855 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0810 04:08:16.905239 (+ 384us) server_negotiation.cc:299] Negotiation successful
0810 04:08:16.905326 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260810 04:08:16.905954 12242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:08:16.894615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:34735 (local address 127.31.32.3:39951)
0810 04:08:16.895250 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0810 04:08:16.895292 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 04:08:16.895927 (+ 635us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:08:16.896360 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:08:16.896364 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:08:16.896611 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:08:16.896876 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:08:16.896881 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:08:16.897904 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:08:16.897905 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:08:16.899151 (+ 1246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:08:16.899154 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:08:16.899684 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:08:16.899685 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:08:16.899840 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:08:16.900450 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:08:16.900460 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:08:16.902035 (+ 1575us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:08:16.903942 (+ 1907us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:08:16.903949 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:08:16.903951 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:08:16.904245 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:08:16.904474 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:08:16.904475 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:08:16.904475 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:08:16.904603 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:08:16.904886 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:08:16.904891 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:08:16.905079 (+ 188us) client_negotiation.cc:769] Sending connection context
0810 04:08:16.905566 (+ 487us) client_negotiation.cc:240] Negotiation successful
0810 04:08:16.905723 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":597,"thread_start_us":170,"threads_started":1}
I20260810 04:08:16.906386 12240 heartbeater.cc:347] Connected to a master server at 127.31.32.62:34735
I20260810 04:08:16.906481 12240 heartbeater.cc:464] Registering TS with master...
I20260810 04:08:16.906670 12240 heartbeater.cc:511] Master 127.31.32.62:34735 requested a full tablet report, sending...
I20260810 04:08:16.907009 11702 ts_manager.cc:194] Registered new tserver with Master: 9fccc96ef747484bb7b8ddfbc7c04c75 (127.31.32.3:43187)
I20260810 04:08:16.907508 11702 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.32.3@KRBTEST.COM'} at 127.31.32.3:39951
I20260810 04:08:16.912910 31872 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 04:08:16.921845 11935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:08:16.914581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55270 (local address 127.31.32.62:34735)
0810 04:08:16.914622 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0810 04:08:16.914625 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:08:16.915183 (+ 558us) server_negotiation.cc:407] Connection header received
0810 04:08:16.915212 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:08:16.915212 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:08:16.915225 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:08:16.915309 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:08:16.915862 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:08:16.916682 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:08:16.917438 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:08:16.917728 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:08:16.917810 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:08:16.919079 (+ 1269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:08:16.919081 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:08:16.919081 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:08:16.919088 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:08:16.920738 (+ 1650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:08:16.921098 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:08:16.921099 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:08:16.921099 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:08:16.921164 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:08:16.921371 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:08:16.921372 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:08:16.921372 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:08:16.921496 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:08:16.921596 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0810 04:08:16.921683 (+ 87us) server_negotiation.cc:299] Negotiation successful
0810 04:08:16.921757 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260810 04:08:16.924676 11702 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55270:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 04:08:16.926965 11702 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 04:08:16.934247 12259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:08:16.930265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60936 (local address 127.31.32.1:34889)
0810 04:08:16.930560 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0810 04:08:16.930564 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 04:08:16.930574 (+ 10us) server_negotiation.cc:407] Connection header received
0810 04:08:16.930608 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:08:16.930611 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:08:16.930625 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:08:16.930705 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:08:16.930828 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:08:16.931862 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:08:16.933106 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:08:16.933777 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:08:16.933886 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:08:16.933950 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0810 04:08:16.934122 (+ 172us) server_negotiation.cc:299] Negotiation successful
0810 04:08:16.934170 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":60,"threads_started":1}
I20260810 04:08:16.934252 12255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:08:16.930034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.1:34889 (local address 127.0.0.1:60936)
0810 04:08:16.930375 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0810 04:08:16.930386 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 04:08:16.930429 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:08:16.930721 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:08:16.930722 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:08:16.930725 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:08:16.930763 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:08:16.930765 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:08:16.931949 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:08:16.931950 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:08:16.933011 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:08:16.933014 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:08:16.933910 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:08:16.933911 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:08:16.933995 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:08:16.934012 (+ 17us) client_negotiation.cc:769] Sending connection context
0810 04:08:16.934092 (+ 80us) client_negotiation.cc:240] Negotiation successful
0810 04:08:16.934143 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":74,"threads_started":1}
I20260810 04:08:16.934267 12253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:08:16.929626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.2:41581 (local address 127.0.0.1:36172)
0810 04:08:16.929977 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0810 04:08:16.930018 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 04:08:16.930090 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:08:16.930413 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:08:16.930417 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:08:16.930419 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:08:16.930487 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:08:16.930489 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:08:16.931783 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:08:16.931793 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:08:16.932874 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:08:16.932877 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:08:16.933999 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:08:16.934000 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:08:16.934063 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:08:16.934087 (+ 24us) client_negotiation.cc:769] Sending connection context
0810 04:08:16.934142 (+ 55us) client_negotiation.cc:240] Negotiation successful
0810 04:08:16.934179 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":96,"threads_started":1}
I20260810 04:08:16.934840 12254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:08:16.929771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36172 (local address 127.31.32.2:41581)
0810 04:08:16.930062 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0810 04:08:16.930068 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 04:08:16.930080 (+ 12us) server_negotiation.cc:407] Connection header received
0810 04:08:16.930220 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:08:16.930223 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:08:16.930241 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:08:16.930317 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:08:16.930569 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:08:16.931691 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:08:16.932984 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:08:16.933743 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:08:16.933886 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:08:16.933972 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0810 04:08:16.934428 (+ 456us) server_negotiation.cc:299] Negotiation successful
0810 04:08:16.934494 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":92,"threads_started":1}
I20260810 04:08:16.935339 11856 tablet_service.cc:1467] Processing CreateTablet for tablet e59693b4d58c4d83b8c451286b58c358 (DEFAULT_TABLE table=test-table [id=973412a8cf6847b0b3ba24332add3118]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:08:16.935623 11856 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e59693b4d58c4d83b8c451286b58c358. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:08:16.935556 12004 tablet_service.cc:1467] Processing CreateTablet for tablet e59693b4d58c4d83b8c451286b58c358 (DEFAULT_TABLE table=test-table [id=973412a8cf6847b0b3ba24332add3118]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:08:16.935895 12004 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e59693b4d58c4d83b8c451286b58c358. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:08:16.938060 12260 tablet_bootstrap.cc:492] T e59693b4d58c4d83b8c451286b58c358 P ec06548e7ccc422e893f208652e3b939: Bootstrap starting.
I20260810 04:08:16.938760 12260 tablet_bootstrap.cc:654] T e59693b4d58c4d83b8c451286b58c358 P ec06548e7ccc422e893f208652e3b939: Neither blocks nor log segments found. Creating new log.
I20260810 04:08:16.939069 12260 log.cc:824] T e59693b4d58c4d83b8c451286b58c358 P ec06548e7ccc422e893f208652e3b939: Log is configured to *not* fsync() on all Append() calls
I20260810 04:08:16.939303 12261 tablet_bootstrap.cc:492] T e59693b4d58c4d83b8c451286b58c358 P 574bf888816c47f69bc82232cfc36ef7: Bootstrap starting.
I20260810 04:08:16.940430 12260 tablet_bootstrap.cc:492] T e59693b4d58c4d83b8c451286b58c358 P ec06548e7ccc422e893f208652e3b939: No bootstrap required, opened a new log
I20260810 04:08:16.940511 12260 ts_tablet_manager.cc:1397] T e59693b4d58c4d83b8c451286b58c358 P ec06548e7ccc422e893f208652e3b939: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260810 04:08:16.940593 12261 tablet_bootstrap.cc:654] T e59693b4d58c4d83b8c451286b58c358 P 574bf888816c47f69bc82232cfc36ef7: Neither blocks nor log segments found. Creating new log.
I20260810 04:08:16.940963 12261 log.cc:824] T e59693b4d58c4d83b8c451286b58c358 P 574bf888816c47f69bc82232cfc36ef7: Log is configured to *not* fsync() on all Append() calls
I20260810 04:08:16.941864 12261 tablet_bootstrap.cc:492] T e59693b4d58c4d83b8c451286b58c358 P 574bf888816c47f69bc82232cfc36ef7: No bootstrap required, opened a new log
I20260810 04:08:16.941928 12261 ts_tablet_manager.cc:1397] T e59693b4d58c4d83b8c451286b58c358 P 574bf888816c47f69bc82232cfc36ef7: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 04:08:16.943491 12260 raft_consensus.cc:348] T e59693b4d58c4d83b8c451286b58c358 P ec06548e7ccc422e893f208652e3b939 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "574bf888816c47f69bc82232cfc36ef7" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 41581 } } peers { permanent_uuid: "9fccc96ef747484bb7b8ddfbc7c04c75" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 43187 } } peers { permanent_uuid: "ec06548e7ccc422e893f208652e3b939" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 34889 } }
I20260810 04:08:16.943647 12260 raft_consensus.cc:374] T e59693b4d58c4d83b8c451286b58c358 P ec06548e7ccc422e893f208652e3b939 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:08:16.943677 12260 raft_consensus.cc:729] T e59693b4d58c4d83b8c451286b58c358 P ec06548e7ccc422e893f208652e3b939 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec06548e7ccc422e893f208652e3b939, State: Initialized, Role: FOLLOWER
I20260810 04:08:16.943677 12256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:08:16.929876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.3:43187 (local address 127.0.0.1:37048)
0810 04:08:16.930178 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0810 04:08:16.930189 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 04:08:16.930246 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:08:16.939967 (+ 9721us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:08:16.939969 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:08:16.939971 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:08:16.940008 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:08:16.940010 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:08:16.941241 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:08:16.941242 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:08:16.942199 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:08:16.942202 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:08:16.943425 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:08:16.943427 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:08:16.943496 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:08:16.943501 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 04:08:16.943534 (+ 33us) client_negotiation.cc:240] Negotiation successful
0810 04:08:16.943579 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":65,"threads_started":1}
I20260810 04:08:16.943907 12260 consensus_queue.cc:260] T e59693b4d58c4d83b8c451286b58c358 P ec06548e7ccc422e893f208652e3b939 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "574bf888816c47f69bc82232cfc36ef7" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 41581 } } peers { permanent_uuid: "9fccc96ef747484bb7b8ddfbc7c04c75" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 43187 } } peers { permanent_uuid: "ec06548e7ccc422e893f208652e3b939" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 34889 } }
I20260810 04:08:16.943827 12261 raft_consensus.cc:348] T e59693b4d58c4d83b8c451286b58c358 P 574bf888816c47f69bc82232cfc36ef7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "574bf888816c47f69bc82232cfc36ef7" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 41581 } } peers { permanent_uuid: "9fccc96ef747484bb7b8ddfbc7c04c75" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 43187 } } peers { permanent_uuid: "ec06548e7ccc422e893f208652e3b939" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 34889 } }
I20260810 04:08:16.943992 12261 raft_consensus.cc:374] T e59693b4d58c4d83b8c451286b58c358 P 574bf888816c47f69bc82232cfc36ef7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:08:16.944031 12261 raft_consensus.cc:729] T e59693b4d58c4d83b8c451286b58c358 P 574bf888816c47f69bc82232cfc36ef7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 574bf888816c47f69bc82232cfc36ef7, State: Initialized, Role: FOLLOWER
I20260810 04:08:16.944068 12257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:08:16.929995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37048 (local address 127.31.32.3:43187)
0810 04:08:16.939756 (+ 9761us) server_negotiation.cc:206] Beginning negotiation
0810 04:08:16.939762 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 04:08:16.939776 (+ 14us) server_negotiation.cc:407] Connection header received
0810 04:08:16.939838 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:08:16.939840 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:08:16.939857 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:08:16.939957 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:08:16.940107 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:08:16.941141 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:08:16.942326 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:08:16.943301 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:08:16.943796 (+ 495us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:08:16.943899 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0810 04:08:16.943930 (+ 31us) server_negotiation.cc:299] Negotiation successful
0810 04:08:16.943972 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9697,"thread_start_us":168,"threads_started":1}
I20260810 04:08:16.944190 12260 ts_tablet_manager.cc:1428] T e59693b4d58c4d83b8c451286b58c358 P ec06548e7ccc422e893f208652e3b939: Time spent starting tablet: real 0.004s user 0.000s sys 0.004s
I20260810 04:08:16.944161 12261 consensus_queue.cc:260] T e59693b4d58c4d83b8c451286b58c358 P 574bf888816c47f69bc82232cfc36ef7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "574bf888816c47f69bc82232cfc36ef7" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 41581 } } peers { permanent_uuid: "9fccc96ef747484bb7b8ddfbc7c04c75" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 43187 } } peers { permanent_uuid: "ec06548e7ccc422e893f208652e3b939" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 34889 } }
I20260810 04:08:16.944335 11933 heartbeater.cc:503] Master 127.31.32.62:34735 was elected leader, sending a full tablet report...
I20260810 04:08:16.944412 12261 ts_tablet_manager.cc:1428] T e59693b4d58c4d83b8c451286b58c358 P 574bf888816c47f69bc82232cfc36ef7: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260810 04:08:16.944538 12086 heartbeater.cc:503] Master 127.31.32.62:34735 was elected leader, sending a full tablet report...
I20260810 04:08:16.944715 12163 tablet_service.cc:1467] Processing CreateTablet for tablet e59693b4d58c4d83b8c451286b58c358 (DEFAULT_TABLE table=test-table [id=973412a8cf6847b0b3ba24332add3118]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:08:16.945014 12163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e59693b4d58c4d83b8c451286b58c358. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:08:16.947669 12267 tablet_bootstrap.cc:492] T e59693b4d58c4d83b8c451286b58c358 P 9fccc96ef747484bb7b8ddfbc7c04c75: Bootstrap starting.
I20260810 04:08:16.948509 12267 tablet_bootstrap.cc:654] T e59693b4d58c4d83b8c451286b58c358 P 9fccc96ef747484bb7b8ddfbc7c04c75: Neither blocks nor log segments found. Creating new log.
I20260810 04:08:16.948788 12267 log.cc:824] T e59693b4d58c4d83b8c451286b58c358 P 9fccc96ef747484bb7b8ddfbc7c04c75: Log is configured to *not* fsync() on all Append() calls
I20260810 04:08:16.949410 12267 tablet_bootstrap.cc:492] T e59693b4d58c4d83b8c451286b58c358 P 9fccc96ef747484bb7b8ddfbc7c04c75: No bootstrap required, opened a new log
I20260810 04:08:16.949461 12267 ts_tablet_manager.cc:1397] T e59693b4d58c4d83b8c451286b58c358 P 9fccc96ef747484bb7b8ddfbc7c04c75: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 04:08:16.951000 12267 raft_consensus.cc:348] T e59693b4d58c4d83b8c451286b58c358 P 9fccc96ef747484bb7b8ddfbc7c04c75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "574bf888816c47f69bc82232cfc36ef7" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 41581 } } peers { permanent_uuid: "9fccc96ef747484bb7b8ddfbc7c04c75" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 43187 } } peers { permanent_uuid: "ec06548e7ccc422e893f208652e3b939" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 34889 } }
I20260810 04:08:16.951116 12267 raft_consensus.cc:374] T e59693b4d58c4d83b8c451286b58c358 P 9fccc96ef747484bb7b8ddfbc7c04c75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:08:16.951138 12267 raft_consensus.cc:729] T e59693b4d58c4d83b8c451286b58c358 P 9fccc96ef747484bb7b8ddfbc7c04c75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9fccc96ef747484bb7b8ddfbc7c04c75, State: Initialized, Role: FOLLOWER
I20260810 04:08:16.951241 12267 consensus_queue.cc:260] T e59693b4d58c4d83b8c451286b58c358 P 9fccc96ef747484bb7b8ddfbc7c04c75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "574bf888816c47f69bc82232cfc36ef7" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 41581 } } peers { permanent_uuid: "9fccc96ef747484bb7b8ddfbc7c04c75" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 43187 } } peers { permanent_uuid: "ec06548e7ccc422e893f208652e3b939" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 34889 } }
I20260810 04:08:16.951493 12267 ts_tablet_manager.cc:1428] T e59693b4d58c4d83b8c451286b58c358 P 9fccc96ef747484bb7b8ddfbc7c04c75: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260810 04:08:16.951604 12240 heartbeater.cc:503] Master 127.31.32.62:34735 was elected leader, sending a full tablet report...
I20260810 04:08:17.022614 12266 raft_consensus.cc:482] T e59693b4d58c4d83b8c451286b58c358 P 574bf888816c47f69bc82232cfc36ef7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 04:08:17.022768 12266 raft_consensus.cc:504] T e59693b4d58c4d83b8c451286b58c358 P 574bf888816c47f69bc82232cfc36ef7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "574bf888816c47f69bc82232cfc36ef7" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 41581 } } peers { permanent_uuid: "9fccc96ef747484bb7b8ddfbc7c04c75" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 43187 } } peers { permanent_uuid: "ec06548e7ccc422e893f208652e3b939" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 34889 } }
I20260810 04:08:17.023170 12266 leader_election.cc:288] T e59693b4d58c4d83b8c451286b58c358 P 574bf888816c47f69bc82232cfc36ef7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9fccc96ef747484bb7b8ddfbc7c04c75 (127.31.32.3:43187), ec06548e7ccc422e893f208652e3b939 (127.31.32.1:34889)
I20260810 04:08:17.027526 12088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:08:17.023393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.3:43187 (local address 127.31.32.2:60193)
0810 04:08:17.023683 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0810 04:08:17.023694 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 04:08:17.023738 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:08:17.024170 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:08:17.024171 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:08:17.024173 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:08:17.024245 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:08:17.024247 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:08:17.025254 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:08:17.025255 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:08:17.026515 (+ 1260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:08:17.026517 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:08:17.027281 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:08:17.027282 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:08:17.027359 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:08:17.027363 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 04:08:17.027408 (+ 45us) client_negotiation.cc:240] Negotiation successful
0810 04:08:17.027445 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272}
I20260810 04:08:17.027917 12257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:08:17.023880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.2:60193 (local address 127.31.32.3:43187)
0810 04:08:17.023923 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0810 04:08:17.023925 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:08:17.023932 (+ 7us) server_negotiation.cc:407] Connection header received
0810 04:08:17.023960 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:08:17.023961 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:08:17.023971 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:08:17.024016 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:08:17.024354 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:08:17.025137 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:08:17.026616 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:08:17.027174 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:08:17.027239 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:08:17.027250 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0810 04:08:17.027804 (+ 554us) server_negotiation.cc:299] Negotiation successful
0810 04:08:17.027854 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260810 04:08:17.028203 12184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e59693b4d58c4d83b8c451286b58c358" candidate_uuid: "574bf888816c47f69bc82232cfc36ef7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9fccc96ef747484bb7b8ddfbc7c04c75" is_pre_election: true
I20260810 04:08:17.028358 12184 raft_consensus.cc:2454] T e59693b4d58c4d83b8c451286b58c358 P 9fccc96ef747484bb7b8ddfbc7c04c75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 574bf888816c47f69bc82232cfc36ef7 in term 0.
I20260810 04:08:17.028585 11963 leader_election.cc:302] T e59693b4d58c4d83b8c451286b58c358 P 574bf888816c47f69bc82232cfc36ef7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 574bf888816c47f69bc82232cfc36ef7, 9fccc96ef747484bb7b8ddfbc7c04c75; no voters:
I20260810 04:08:17.028730 12266 raft_consensus.cc:2781] T e59693b4d58c4d83b8c451286b58c358 P 574bf888816c47f69bc82232cfc36ef7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 04:08:17.028777 12266 raft_consensus.cc:482] T e59693b4d58c4d83b8c451286b58c358 P 574bf888816c47f69bc82232cfc36ef7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 04:08:17.028803 12266 raft_consensus.cc:3037] T e59693b4d58c4d83b8c451286b58c358 P 574bf888816c47f69bc82232cfc36ef7 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:08:17.030486 12266 raft_consensus.cc:504] T e59693b4d58c4d83b8c451286b58c358 P 574bf888816c47f69bc82232cfc36ef7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "574bf888816c47f69bc82232cfc36ef7" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 41581 } } peers { permanent_uuid: "9fccc96ef747484bb7b8ddfbc7c04c75" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 43187 } } peers { permanent_uuid: "ec06548e7ccc422e893f208652e3b939" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 34889 } }
I20260810 04:08:17.030655 12266 leader_election.cc:288] T e59693b4d58c4d83b8c451286b58c358 P 574bf888816c47f69bc82232cfc36ef7 [CANDIDATE]: Term 1 election: Requested vote from peers 9fccc96ef747484bb7b8ddfbc7c04c75 (127.31.32.3:43187), ec06548e7ccc422e893f208652e3b939 (127.31.32.1:34889)
I20260810 04:08:17.030990 12088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:08:17.023484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.1:34889 (local address 127.31.32.2:55249)
0810 04:08:17.027610 (+ 4126us) negotiation.cc:107] Waiting for socket to connect
0810 04:08:17.027620 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 04:08:17.027672 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:08:17.027878 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:08:17.027879 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:08:17.027881 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:08:17.027916 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:08:17.027917 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:08:17.028812 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:08:17.028813 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:08:17.029873 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:08:17.029875 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:08:17.030754 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:08:17.030755 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:08:17.030830 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:08:17.030834 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 04:08:17.030882 (+ 48us) client_negotiation.cc:240] Negotiation successful
0810 04:08:17.030917 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4116,"thread_start_us":105,"threads_started":1}
I20260810 04:08:17.030948 12184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e59693b4d58c4d83b8c451286b58c358" candidate_uuid: "574bf888816c47f69bc82232cfc36ef7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9fccc96ef747484bb7b8ddfbc7c04c75"
I20260810 04:08:17.031038 12184 raft_consensus.cc:3037] T e59693b4d58c4d83b8c451286b58c358 P 9fccc96ef747484bb7b8ddfbc7c04c75 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:08:17.031286 12259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:08:17.024100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.2:55249 (local address 127.31.32.1:34889)
0810 04:08:17.024356 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0810 04:08:17.024359 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:08:17.027669 (+ 3310us) server_negotiation.cc:407] Connection header received
0810 04:08:17.027726 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:08:17.027727 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:08:17.027742 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:08:17.027821 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:08:17.027999 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:08:17.028705 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:08:17.029974 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:08:17.030637 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:08:17.030711 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:08:17.030722 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0810 04:08:17.031148 (+ 426us) server_negotiation.cc:299] Negotiation successful
0810 04:08:17.031198 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":226}
I20260810 04:08:17.031561 11876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e59693b4d58c4d83b8c451286b58c358" candidate_uuid: "574bf888816c47f69bc82232cfc36ef7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec06548e7ccc422e893f208652e3b939" is_pre_election: true
I20260810 04:08:17.031694 11876 raft_consensus.cc:2454] T e59693b4d58c4d83b8c451286b58c358 P ec06548e7ccc422e893f208652e3b939 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 574bf888816c47f69bc82232cfc36ef7 in term 0.
I20260810 04:08:17.031822 11860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e59693b4d58c4d83b8c451286b58c358" candidate_uuid: "574bf888816c47f69bc82232cfc36ef7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec06548e7ccc422e893f208652e3b939"
I20260810 04:08:17.031883 11860 raft_consensus.cc:3037] T e59693b4d58c4d83b8c451286b58c358 P ec06548e7ccc422e893f208652e3b939 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:08:17.032053 12184 raft_consensus.cc:2454] T e59693b4d58c4d83b8c451286b58c358 P 9fccc96ef747484bb7b8ddfbc7c04c75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 574bf888816c47f69bc82232cfc36ef7 in term 1.
I20260810 04:08:17.032600 11963 leader_election.cc:302] T e59693b4d58c4d83b8c451286b58c358 P 574bf888816c47f69bc82232cfc36ef7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 574bf888816c47f69bc82232cfc36ef7, 9fccc96ef747484bb7b8ddfbc7c04c75; no voters:
I20260810 04:08:17.032976 12266 raft_consensus.cc:2781] T e59693b4d58c4d83b8c451286b58c358 P 574bf888816c47f69bc82232cfc36ef7 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:08:17.033144 12266 raft_consensus.cc:686] T e59693b4d58c4d83b8c451286b58c358 P 574bf888816c47f69bc82232cfc36ef7 [term 1 LEADER]: Becoming Leader. State: Replica: 574bf888816c47f69bc82232cfc36ef7, State: Running, Role: LEADER
I20260810 04:08:17.033241 12266 consensus_queue.cc:237] T e59693b4d58c4d83b8c451286b58c358 P 574bf888816c47f69bc82232cfc36ef7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "574bf888816c47f69bc82232cfc36ef7" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 41581 } } peers { permanent_uuid: "9fccc96ef747484bb7b8ddfbc7c04c75" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 43187 } } peers { permanent_uuid: "ec06548e7ccc422e893f208652e3b939" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 34889 } }
I20260810 04:08:17.033408 11860 raft_consensus.cc:2454] T e59693b4d58c4d83b8c451286b58c358 P ec06548e7ccc422e893f208652e3b939 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 574bf888816c47f69bc82232cfc36ef7 in term 1.
I20260810 04:08:17.034451 11702 catalog_manager.cc:5495] T e59693b4d58c4d83b8c451286b58c358 P 574bf888816c47f69bc82232cfc36ef7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 574bf888816c47f69bc82232cfc36ef7 (127.31.32.2). New cstate: current_term: 1 leader_uuid: "574bf888816c47f69bc82232cfc36ef7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "574bf888816c47f69bc82232cfc36ef7" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 41581 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9fccc96ef747484bb7b8ddfbc7c04c75" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 43187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec06548e7ccc422e893f208652e3b939" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 34889 } health_report { overall_health: UNKNOWN } } }
I20260810 04:08:17.051292 12254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:08:17.047375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36174 (local address 127.31.32.2:41581)
0810 04:08:17.047429 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0810 04:08:17.047432 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:08:17.047478 (+ 46us) server_negotiation.cc:407] Connection header received
0810 04:08:17.047569 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:08:17.047570 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:08:17.047589 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:08:17.047679 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 04:08:17.048297 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:08:17.049119 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:08:17.050270 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:08:17.050613 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:08:17.050676 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:08:17.050903 (+ 227us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 04:08:17.051035 (+ 132us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 04:08:17.051148 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0810 04:08:17.051167 (+ 19us) server_negotiation.cc:299] Negotiation successful
0810 04:08:17.051188 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260810 04:08:17.053148 11860 raft_consensus.cc:1261] T e59693b4d58c4d83b8c451286b58c358 P ec06548e7ccc422e893f208652e3b939 [term 1 FOLLOWER]: Refusing update from remote peer 574bf888816c47f69bc82232cfc36ef7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:08:17.053148 12184 raft_consensus.cc:1261] T e59693b4d58c4d83b8c451286b58c358 P 9fccc96ef747484bb7b8ddfbc7c04c75 [term 1 FOLLOWER]: Refusing update from remote peer 574bf888816c47f69bc82232cfc36ef7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:08:17.053500 12271 consensus_queue.cc:1035] T e59693b4d58c4d83b8c451286b58c358 P 574bf888816c47f69bc82232cfc36ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9fccc96ef747484bb7b8ddfbc7c04c75" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 43187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:08:17.053668 12266 consensus_queue.cc:1035] T e59693b4d58c4d83b8c451286b58c358 P 574bf888816c47f69bc82232cfc36ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec06548e7ccc422e893f208652e3b939" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 34889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:08:17.057662 12275 mvcc.cc:205] Tried to move back new op lower bound from 7316827738327334912 to 7316827738248826880. Current Snapshot: MvccSnapshot[applied={T|T < 7316827738327334912}]
I20260810 04:08:17.058260 12276 mvcc.cc:205] Tried to move back new op lower bound from 7316827738327334912 to 7316827738248826880. Current Snapshot: MvccSnapshot[applied={T|T < 7316827738327334912}]
I20260810 04:08:17.058709 12277 mvcc.cc:205] Tried to move back new op lower bound from 7316827738327334912 to 7316827738248826880. Current Snapshot: MvccSnapshot[applied={T|T < 7316827738327334912}]
I20260810 04:08:17.061067 11702 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55270:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 04:08:17.061182 11702 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55270:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 04:08:17.061846 11702 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f1e5112d255148e18c83e49f537a5be6: Sending DeleteTablet for 3 replicas of tablet e59693b4d58c4d83b8c451286b58c358
I20260810 04:08:17.062193 12004 tablet_service.cc:1514] Processing DeleteTablet for tablet e59693b4d58c4d83b8c451286b58c358 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:08:17 UTC) from {username='kudu', principal='kudu/127.31.32.62@KRBTEST.COM'} at 127.0.0.1:36172
I20260810 04:08:17.062402 12163 tablet_service.cc:1514] Processing DeleteTablet for tablet e59693b4d58c4d83b8c451286b58c358 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:08:17 UTC) from {username='kudu', principal='kudu/127.31.32.62@KRBTEST.COM'} at 127.0.0.1:37048
I20260810 04:08:17.062692 12285 tablet_replica.cc:330] T e59693b4d58c4d83b8c451286b58c358 P 9fccc96ef747484bb7b8ddfbc7c04c75: stopping tablet replica
I20260810 04:08:17.062686 11856 tablet_service.cc:1514] Processing DeleteTablet for tablet e59693b4d58c4d83b8c451286b58c358 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:08:17 UTC) from {username='kudu', principal='kudu/127.31.32.62@KRBTEST.COM'} at 127.0.0.1:60936
I20260810 04:08:17.062849 12285 raft_consensus.cc:2229] T e59693b4d58c4d83b8c451286b58c358 P 9fccc96ef747484bb7b8ddfbc7c04c75 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 04:08:17.063026 12285 raft_consensus.cc:2258] T e59693b4d58c4d83b8c451286b58c358 P 9fccc96ef747484bb7b8ddfbc7c04c75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:08:17.063225 31872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskws89Io/build/release/bin/kudu with pid 11790
I20260810 04:08:17.064244 12285 ts_tablet_manager.cc:1905] T e59693b4d58c4d83b8c451286b58c358 P 9fccc96ef747484bb7b8ddfbc7c04c75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:08:17.066361 12285 ts_tablet_manager.cc:1918] T e59693b4d58c4d83b8c451286b58c358 P 9fccc96ef747484bb7b8ddfbc7c04c75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 04:08:17.066527 12285 log.cc:1196] T e59693b4d58c4d83b8c451286b58c358 P 9fccc96ef747484bb7b8ddfbc7c04c75: Deleting WAL directory at /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786334794676589-31872-0/minicluster-data/ts-2/wal/wals/e59693b4d58c4d83b8c451286b58c358
I20260810 04:08:17.067413 12285 ts_tablet_manager.cc:1939] T e59693b4d58c4d83b8c451286b58c358 P 9fccc96ef747484bb7b8ddfbc7c04c75: Deleting consensus metadata
I20260810 04:08:17.068006 11683 catalog_manager.cc:4841] TS 9fccc96ef747484bb7b8ddfbc7c04c75 (127.31.32.3:43187): tablet e59693b4d58c4d83b8c451286b58c358 (table test-table [id=973412a8cf6847b0b3ba24332add3118]) successfully deleted
I20260810 04:08:17.079672 12287 tablet_replica.cc:330] T e59693b4d58c4d83b8c451286b58c358 P 574bf888816c47f69bc82232cfc36ef7: stopping tablet replica
I20260810 04:08:17.079823 12287 raft_consensus.cc:2229] T e59693b4d58c4d83b8c451286b58c358 P 574bf888816c47f69bc82232cfc36ef7 [term 1 LEADER]: Raft consensus shutting down.
I20260810 04:08:17.079972 12287 raft_consensus.cc:2258] T e59693b4d58c4d83b8c451286b58c358 P 574bf888816c47f69bc82232cfc36ef7 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260810 04:08:17.082474 11683 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.32.1:34889 (error 108)
I20260810 04:08:17.083190 31872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskws89Io/build/release/bin/kudu with pid 11944
I20260810 04:08:17.097249 12256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:08:17.097057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.1:34889 (local address 127.0.0.1:60938)
0810 04:08:17.097120 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0810 04:08:17.097193 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.1:34889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260810 04:08:17.097373 31872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskws89Io/build/release/bin/kudu with pid 12092
W20260810 04:08:17.098353 11683 catalog_manager.cc:4584] TS ec06548e7ccc422e893f208652e3b939 (127.31.32.1:34889): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e59693b4d58c4d83b8c451286b58c358: Network error: Client connection negotiation failed: client connection to 127.31.32.1:34889: connect: Connection refused (error 111)
I20260810 04:08:17.098721 12256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:08:17.098627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.2:41581 (local address 127.0.0.1:36176)
0810 04:08:17.098671 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0810 04:08:17.098694 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.2:41581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260810 04:08:17.105129 31872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskws89Io/build/release/bin/kudu with pid 11666
I20260810 04:08:17.115355 31872 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260810 04:08:17.459450 31872 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260810 04:08:17.459581 31872 mini_ranger.cc:66] Stopping Ranger...
I20260810 04:08:17.803716 31872 mini_ranger.cc:68] Stopped Ranger
2026-08-10 04:08:17.803 UTC [11198] LOG: received smart shutdown request
2026-08-10 04:08:17.807 UTC [11198] LOG: background worker "logical replication launcher" (PID 11207) exited with exit code 1
2026-08-10 04:08:17.808 UTC [11202] LOG: shutting down
2026-08-10 04:08:17.982 UTC [11198] LOG: database system is shut down
2026-08-10T04:08:17Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (47336 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 04:08:18 dist-test-slave-8w9v krb5kdc[12299](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 04:08:18 dist-test-slave-8w9v krb5kdc[12299](info): set up 2 sockets
Aug 10 04:08:18 dist-test-slave-8w9v krb5kdc[12299](info): commencing operation
krb5kdc: starting...
W20260810 04:08:19.187395 31872 mini_kdc.cc:121] Time spent starting KDC: real 1.072s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/krb5kdc/test-user.keytab.
Aug 10 04:08:19 dist-test-slave-8w9v krb5kdc[12299](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334899, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T04:08:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T04:08:19Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.31.32.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.31.32.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.31.32.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/krb5kdc/rangeradmin_127.31.32.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.31.32.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/krb5kdc/rangeradmin_127.31.32.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.31.32.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.31.32.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.31.32.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/krb5kdc/rangerlookup_127.31.32.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.31.32.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/krb5kdc/rangerlookup_127.31.32.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.31.32.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.31.32.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.31.32.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/krb5kdc/HTTP_127.31.32.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.31.32.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/krb5kdc/HTTP_127.31.32.20@KRBTEST.COM.keytab.
I20260810 04:08:19.350431 31872 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskws89Io/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/minicluster-data/postgres -l logfile start
2026-08-10 04:08:21.807 UTC [12332] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-10 04:08:21.807 UTC [12332] LOG: listening on IPv4 address "127.31.32.20", port 44535
2026-08-10 04:08:21.811 UTC [12332] LOG: listening on Unix socket "/tmp/.s.PGSQL.44535"
2026-08-10 04:08:21.832 UTC [12335] LOG: database system was shut down at 2026-08-10 04:08:20 UTC
2026-08-10 04:08:21.836 UTC [12332] LOG: database system is ready to accept connections
I20260810 04:08:22.856642 31872 mini_postgres.cc:96] Postgres bound to 44535
2026-08-10 04:08:22.863 UTC [12344] FATAL: database "slave" does not exist
127.31.32.20:44535 - accepting connections
I20260810 04:08:22.864216 31872 mini_ranger.cc:161] Starting Ranger...
I20260810 04:08:22.880115 31872 mini_ranger.cc:84] Created miniranger Postgres user
I20260810 04:08:23.346498 31872 mini_ranger.cc:87] Created ranger Postgres database
I20260810 04:08:23.346612 31872 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/minicluster-data/ranger-admin
2026-08-10 04:08:23,608 [I] DB FLAVOR :POSTGRES
2026-08-10 04:08:23,609 [I] --------- Verifying Ranger DB connection ---------
2026-08-10 04:08:23,609 [I] Checking connection..
2026-08-10 04:08:23,609 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskws89Io/build/release/bin/postgresql.jar:/tmp/dist-test-taskws89Io/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.32.20:44535/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-10 04:08:23,859 [I] Checking connection passed.
2026-08-10 04:08:23,860 [I] --------- Verifying version history table ---------
2026-08-10 04:08:23,860 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskws89Io/build/release/bin/postgresql.jar:/tmp/dist-test-taskws89Io/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.32.20:44535/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 04:08:24,135 [I] Table x_db_version_h does not exist in database ranger
2026-08-10 04:08:24,135 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskws89Io/build/release/bin/postgresql.jar:/tmp/dist-test-taskws89Io/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.32.20:44535/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 04:08:24,387 [I] Table x_db_version_h does not exist in database ranger
2026-08-10 04:08:24,388 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-10 04:08:24,388 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskws89Io/build/release/bin/postgresql.jar:/tmp/dist-test-taskws89Io/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.32.20:44535/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskws89Io/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-10 04:08:24.616 UTC [12417] WARNING: there is no transaction in progress
2026-08-10 04:08:24,623 [I] create_dbversion_catalog.sql file imported successfully
2026-08-10 04:08:24,624 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskws89Io/build/release/bin/postgresql.jar:/tmp/dist-test-taskws89Io/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.32.20:44535/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 04:08:24,884 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-10 04:08:24,884 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-10 04:08:24,884 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskws89Io/build/release/bin/postgresql.jar:/tmp/dist-test-taskws89Io/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.32.20:44535/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-10 04:08:25,139 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskws89Io/build/release/bin/postgresql.jar:/tmp/dist-test-taskws89Io/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.32.20:44535/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-10 04:08:25,390 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskws89Io/build/release/bin/postgresql.jar:/tmp/dist-test-taskws89Io/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.32.20:44535/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-8w9v.c.gcp-upstream.internal','N') ;"
2026-08-10 04:08:25,609 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskws89Io/build/release/bin/postgresql.jar:/tmp/dist-test-taskws89Io/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.32.20:44535/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-10 04:08:25,872 [I] Table x_portal_user does not exist in database ranger
2026-08-10 04:08:25,873 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskws89Io/build/release/bin/postgresql.jar:/tmp/dist-test-taskws89Io/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.32.20:44535/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-10 04:08:26,141 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-10 04:08:26,142 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskws89Io/build/release/bin/postgresql.jar:/tmp/dist-test-taskws89Io/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.32.20:44535/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-10 04:08:26,399 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskws89Io/build/release/bin/postgresql.jar:/tmp/dist-test-taskws89Io/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.32.20:44535/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-10 04:08:26,652 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-10 04:08:26,653 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskws89Io/build/release/bin/postgresql.jar:/tmp/dist-test-taskws89Io/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.32.20:44535/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskws89Io/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-10 04:08:27.560 UTC [12538] WARNING: there is no transaction in progress
2026-08-10 04:08:27.576 UTC [12538] WARNING: there is no transaction in progress
2026-08-10 04:08:27.591 UTC [12538] WARNING: there is no transaction in progress
2026-08-10 04:08:27.605 UTC [12538] WARNING: there is no transaction in progress
2026-08-10 04:08:27.624 UTC [12538] WARNING: there is no transaction in progress
2026-08-10 04:08:27.644 UTC [12538] WARNING: there is no transaction in progress
2026-08-10 04:08:27.699 UTC [12538] WARNING: there is no transaction in progress
2026-08-10 04:08:27.706 UTC [12538] WARNING: there is no transaction in progress
2026-08-10 04:08:27.720 UTC [12538] WARNING: there is no transaction in progress
2026-08-10 04:08:27.730 UTC [12538] WARNING: there is no transaction in progress
2026-08-10 04:08:27.741 UTC [12538] WARNING: there is no transaction in progress
2026-08-10 04:08:27.756 UTC [12538] WARNING: there is no transaction in progress
2026-08-10 04:08:27.766 UTC [12538] WARNING: there is no transaction in progress
2026-08-10 04:08:27.771 UTC [12538] WARNING: there is no transaction in progress
2026-08-10 04:08:27.778 UTC [12538] WARNING: there is no transaction in progress
2026-08-10 04:08:27.784 UTC [12538] WARNING: there is no transaction in progress
2026-08-10 04:08:28,391 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-10 04:08:28,392 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskws89Io/build/release/bin/postgresql.jar:/tmp/dist-test-taskws89Io/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.32.20:44535/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-10 04:08:28,634 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-10 04:08:28,634 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskws89Io/build/release/bin/postgresql.jar:/tmp/dist-test-taskws89Io/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.32.20:44535/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-10 04:08:28,895 [I] Table x_portal_user already exists in database 'ranger'
2026-08-10 04:08:28,895 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskws89Io/build/release/bin/postgresql.jar:/tmp/dist-test-taskws89Io/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.32.20:44535/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-10 04:08:29,131 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-10 04:08:29,131 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskws89Io/build/release/bin/postgresql.jar:/tmp/dist-test-taskws89Io/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.32.20:44535/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-10 04:08:29,397 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskws89Io/build/release/bin/postgresql.jar:/tmp/dist-test-taskws89Io/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.32.20:44535/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-10 04:08:29,652 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskws89Io/build/release/bin/postgresql.jar:/tmp/dist-test-taskws89Io/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.32.20:44535/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-8w9v.c.gcp-upstream.internal';"
2026-08-10 04:08:29,893 [I] CORE_DB_SCHEMA import status has been updated
2026-08-10 04:08:29,893 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskws89Io/build/release/bin/postgresql.jar:/tmp/dist-test-taskws89Io/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.32.20:44535/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-10 04:08:30,130 [I] DB_PATCHES have already been applied
I20260810 04:08:30.136171 31872 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/minicluster-data/ranger-admin:/tmp/dist-test-taskws89Io/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskws89Io/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskws89Io/build/release/bin/postgresql.jar:/tmp/dist-test-taskws89Io/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260810 04:08:30.136250 31872 mini_ranger.cc:193] Using host: 127.31.32.20
I20260810 04:08:30.140192 31872 mini_ranger.cc:227] Ranger admin URL: http://127.31.32.20:41921
Aug 10, 2026 4:08:30 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 10, 2026 4:08:30 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 10, 2026 4:08:30 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/minicluster-data/ranger-admin/ews/webapp
Aug 10, 2026 4:08:30 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 10, 2026 4:08:30 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 10, 2026 4:08:30 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/minicluster-data/ranger-admin/ews/webapp].
Aug 10, 2026 4:08:30 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260810 04:08:31.313716 31872 mini_ranger.cc:160] Time spent starting Ranger: real 8.449s user 0.005s sys 0.005s
Aug 10, 2026 4:08:50 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 10, 2026 4:08:50 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Aug 10, 2026 4:08:50 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 10, 2026 4:08:50 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 10, 2026 4:08:50 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 10, 2026 4:08:50 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 10, 2026 4:08:50 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 10, 2026 4:08:50 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 10, 2026 4:08:50 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 10, 2026 4:08:50 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 10, 2026 4:08:50 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 10, 2026 4:08:50 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 10, 2026 4:08:50 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 10, 2026 4:08:50 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 10, 2026 4:08:50 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 10, 2026 4:08:50 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 10, 2026 4:08:50 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 10, 2026 4:08:51 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260810 04:08:51.735661 31872 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.31.32.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.31.32.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.31.32.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/krb5kdc/rangerkms_127.31.32.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.31.32.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/krb5kdc/rangerkms_127.31.32.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.31.32.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.31.32.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.31.32.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/krb5kdc/HTTP_127.31.32.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.31.32.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/krb5kdc/HTTP_127.31.32.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 10 04:08:51 dist-test-slave-8w9v krb5kdc[12299](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334931, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260810 04:08:51.825371 31872 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260810 04:08:51.835680 31872 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260810 04:08:52.645110 31872 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260810 04:08:52.645200 31872 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/minicluster-data/ranger-kms
I20260810 04:08:52.645219 31872 mini_ranger_kms.cc:227] Using postgres at 127.31.32.20:44535
2026-08-10 04:08:52,833 [I] DB FLAVOR :POSTGRES
2026-08-10 04:08:52,833 [I] --------- Verifying Ranger DB connection ---------
2026-08-10 04:08:52,833 [I] Checking connection
2026-08-10 04:08:52,834 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskws89Io/build/release/bin/postgresql.jar:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.32.20:44535/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-10 04:08:53,093 [I] connection success
2026-08-10 04:08:53,094 [I] --------- Verifying Ranger DB tables ---------
2026-08-10 04:08:53,094 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-10 04:08:53,094 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskws89Io/build/release/bin/postgresql.jar:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.32.20:44535/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-10 04:08:53,336 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-10 04:08:53,336 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-10 04:08:53,336 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-10 04:08:53,337 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskws89Io/build/release/bin/postgresql.jar:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.32.20:44535/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-10 04:08:53,584 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260810 04:08:54.038028 31872 mini_ranger_kms.cc:310] Created kms service
I20260810 04:08:54.254410 31872 mini_ranger_kms.cc:326] Created kudu user
I20260810 04:08:54.301710 31872 mini_ranger_kms.cc:343] Created rangerkms user
Aug 10 04:08:54 dist-test-slave-8w9v krb5kdc[12299](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334931, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.31.32.20@KRBTEST.COM
I20260810 04:08:54.677306 31872 mini_ranger_kms.cc:384] Added ranger policy
I20260810 04:08:54.677454 31872 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/minicluster-data/ranger-kms:/tmp/dist-test-taskws89Io/build/release/bin/postgresql.jar:/tmp/dist-test-taskws89Io/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskws89Io/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260810 04:08:54.677474 31872 mini_ranger_kms.cc:240] Using host: 127.31.32.20
I20260810 04:08:54.681617 31872 mini_ranger_kms.cc:276] Ranger KMS PID: 12739
I20260810 04:08:54.681704 31872 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.31.32.20:53989
Aug 10, 2026 4:08:55 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 10, 2026 4:08:55 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 10, 2026 4:08:55 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 10, 2026 4:08:55 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 10, 2026 4:08:56 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/minicluster-data/ranger-kms/ews/webapp].
Aug 10, 2026 4:08:56 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260810 04:08:56.975625 31872 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.150s user 0.006s sys 0.167s
I20260810 04:08:56.975736 31872 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260810 04:08:56.975793 31872 mini_ranger_kms.cc:401] 127.31.32.20:53989/kms/v1/keys
Aug 10 04:08:57 dist-test-slave-8w9v krb5kdc[12299](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786334937, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.31.32.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Aug 10 04:08:57 dist-test-slave-8w9v krb5kdc[12299](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334937, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.31.32.20@KRBTEST.COM for HTTP/127.31.32.20@KRBTEST.COM
Aug 10 04:08:57 dist-test-slave-8w9v krb5kdc[12299](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334937, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.31.32.20@KRBTEST.COM for HTTP/127.31.32.20@KRBTEST.COM
Aug 10 04:09:00 dist-test-slave-8w9v krb5kdc[12299](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334940, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.31.32.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.32.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.32.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.62.keytab.
Entry for principal kudu/127.31.32.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.62.keytab.
I20260810 04:09:00.144694 31872 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskws89Io/build/release/bin/kudu
/tmp/dist-test-taskws89Io/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.32.62:38215
--webserver_interface=127.31.32.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.62.keytab
--principal=kudu/127.31.32.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.32.20:33429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.32.62:38215
--ranger_config_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.32.20:53989/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:09:00.247706 12793 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:09:00.247915 12793 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:09:00.247938 12793 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:09:00.249069 12793 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:09:00.249105 12793 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:09:00.249119 12793 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:09:00.249132 12793 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:09:00.249145 12793 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:09:00.250665 12793 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.32.20:33429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/minicluster-data/master-0/wal
--ranger_kms_url=127.31.32.20:53989/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.31.32.62:38215
--ranger_config_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.62.keytab
--principal=kudu/127.31.32.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.32.62:38215
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.32.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12793
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:09:00.250865 12793 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:09:00.251037 12793 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:09:00.255771 12798 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:09:00.255810 12799 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:09:00.255823 12801 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:09:00.256251 12793 server_base.cc:1034] running on GCE node
I20260810 04:09:00.256536 12793 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:09:00.256815 12793 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:09:00.257965 12793 hybrid_clock.cc:648] HybridClock initialized: now 1786334940257949 us; error 39 us; skew 500 ppm
Aug 10 04:09:00 dist-test-slave-8w9v krb5kdc[12299](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334940, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:09:00.264219 12793 init.cc:377] Logged in from keytab as kudu/127.31.32.62@KRBTEST.COM (short username kudu)
I20260810 04:09:00.264683 12793 webserver.cc:460] Webserver started at http://127.31.32.62:38119/ using document root <none> and password file <none>
I20260810 04:09:00.264930 12793 fs_manager.cc:359] Metadata directory not provided
I20260810 04:09:00.264969 12793 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:09:00.265125 12793 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 04:09:00 dist-test-slave-8w9v krb5kdc[12299](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334940, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.62@KRBTEST.COM for HTTP/127.31.32.20@KRBTEST.COM
I20260810 04:09:00.310309 12793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/minicluster-data/master-0/data/instance:
uuid: "b615bc77cb4f4ad59550be465d809fb3"
format_stamp: "Formatted at 2026-08-10 04:09:00 on dist-test-slave-8w9v"
server_key: "79a01bdace635be45311ba7ac323fef6"
server_key_iv: "aeca28dbf5cd96d77752a6fe767b02e5"
server_key_version: "kuduclusterkey@0"
I20260810 04:09:00.310799 12793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/minicluster-data/master-0/wal/instance:
uuid: "b615bc77cb4f4ad59550be465d809fb3"
format_stamp: "Formatted at 2026-08-10 04:09:00 on dist-test-slave-8w9v"
server_key: "79a01bdace635be45311ba7ac323fef6"
server_key_iv: "aeca28dbf5cd96d77752a6fe767b02e5"
server_key_version: "kuduclusterkey@0"
I20260810 04:09:00.312376 12793 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260810 04:09:00.323132 12808 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:09:00.323359 12793 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:09:00.323462 12793 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/minicluster-data/master-0/data,/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/minicluster-data/master-0/wal
uuid: "b615bc77cb4f4ad59550be465d809fb3"
format_stamp: "Formatted at 2026-08-10 04:09:00 on dist-test-slave-8w9v"
server_key: "79a01bdace635be45311ba7ac323fef6"
server_key_iv: "aeca28dbf5cd96d77752a6fe767b02e5"
server_key_version: "kuduclusterkey@0"
I20260810 04:09:00.323558 12793 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:09:00.334733 12793 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:09:00.336720 12793 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:09:00.336881 12793 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:09:00.341332 12793 rpc_server.cc:307] RPC server started. Bound to: 127.31.32.62:38215
I20260810 04:09:00.341415 12872 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.32.62:38215 every 8 connection(s)
I20260810 04:09:00.341749 12793 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/minicluster-data/master-0/data/info.pb
I20260810 04:09:00.342505 12873 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:09:00.344713 12873 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b615bc77cb4f4ad59550be465d809fb3: Bootstrap starting.
I20260810 04:09:00.345265 12873 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b615bc77cb4f4ad59550be465d809fb3: Neither blocks nor log segments found. Creating new log.
I20260810 04:09:00.345605 12873 log.cc:824] T 00000000000000000000000000000000 P b615bc77cb4f4ad59550be465d809fb3: Log is configured to *not* fsync() on all Append() calls
I20260810 04:09:00.346278 12873 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b615bc77cb4f4ad59550be465d809fb3: No bootstrap required, opened a new log
I20260810 04:09:00.348322 12873 raft_consensus.cc:348] T 00000000000000000000000000000000 P b615bc77cb4f4ad59550be465d809fb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b615bc77cb4f4ad59550be465d809fb3" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 38215 } }
I20260810 04:09:00.348466 12873 raft_consensus.cc:374] T 00000000000000000000000000000000 P b615bc77cb4f4ad59550be465d809fb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:09:00.348510 12873 raft_consensus.cc:729] T 00000000000000000000000000000000 P b615bc77cb4f4ad59550be465d809fb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b615bc77cb4f4ad59550be465d809fb3, State: Initialized, Role: FOLLOWER
I20260810 04:09:00.348649 12873 consensus_queue.cc:260] T 00000000000000000000000000000000 P b615bc77cb4f4ad59550be465d809fb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b615bc77cb4f4ad59550be465d809fb3" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 38215 } }
I20260810 04:09:00.348724 12873 raft_consensus.cc:388] T 00000000000000000000000000000000 P b615bc77cb4f4ad59550be465d809fb3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 04:09:00.348768 12873 raft_consensus.cc:482] T 00000000000000000000000000000000 P b615bc77cb4f4ad59550be465d809fb3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 04:09:00.348815 12873 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b615bc77cb4f4ad59550be465d809fb3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:09:00.349314 12873 raft_consensus.cc:504] T 00000000000000000000000000000000 P b615bc77cb4f4ad59550be465d809fb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b615bc77cb4f4ad59550be465d809fb3" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 38215 } }
I20260810 04:09:00.349460 12873 leader_election.cc:302] T 00000000000000000000000000000000 P b615bc77cb4f4ad59550be465d809fb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b615bc77cb4f4ad59550be465d809fb3; no voters:
I20260810 04:09:00.349637 12873 leader_election.cc:288] T 00000000000000000000000000000000 P b615bc77cb4f4ad59550be465d809fb3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 04:09:00.349677 12876 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b615bc77cb4f4ad59550be465d809fb3 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:09:00.349860 12876 raft_consensus.cc:686] T 00000000000000000000000000000000 P b615bc77cb4f4ad59550be465d809fb3 [term 1 LEADER]: Becoming Leader. State: Replica: b615bc77cb4f4ad59550be465d809fb3, State: Running, Role: LEADER
I20260810 04:09:00.349890 12873 sys_catalog.cc:564] T 00000000000000000000000000000000 P b615bc77cb4f4ad59550be465d809fb3 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:09:00.349946 12876 consensus_queue.cc:237] T 00000000000000000000000000000000 P b615bc77cb4f4ad59550be465d809fb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b615bc77cb4f4ad59550be465d809fb3" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 38215 } }
I20260810 04:09:00.350363 12877 sys_catalog.cc:455] T 00000000000000000000000000000000 P b615bc77cb4f4ad59550be465d809fb3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b615bc77cb4f4ad59550be465d809fb3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b615bc77cb4f4ad59550be465d809fb3" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 38215 } } }
I20260810 04:09:00.350378 12878 sys_catalog.cc:455] T 00000000000000000000000000000000 P b615bc77cb4f4ad59550be465d809fb3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b615bc77cb4f4ad59550be465d809fb3. Latest consensus state: current_term: 1 leader_uuid: "b615bc77cb4f4ad59550be465d809fb3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b615bc77cb4f4ad59550be465d809fb3" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 38215 } } }
I20260810 04:09:00.350525 12878 sys_catalog.cc:458] T 00000000000000000000000000000000 P b615bc77cb4f4ad59550be465d809fb3 [sys.catalog]: This master's current role is: LEADER
I20260810 04:09:00.350723 12877 sys_catalog.cc:458] T 00000000000000000000000000000000 P b615bc77cb4f4ad59550be465d809fb3 [sys.catalog]: This master's current role is: LEADER
I20260810 04:09:00.350894 12886 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 04:09:00.351001 31872 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskws89Io/build/release/bin/kudu as pid 12793
I20260810 04:09:00.351118 31872 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/minicluster-data/master-0/wal/instance
I20260810 04:09:00.351336 31872 external_mini_cluster.cc:1428] Setting key 538a31f0e44971ce793b9050e909d4dc
I20260810 04:09:00.351410 12886 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 04:09:00.351698 12873 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260810 04:09:00.357173 12886 catalog_manager.cc:1347] Generated new cluster ID: 0dcf7608dc5a4b94a479037fd605a265
I20260810 04:09:00.357234 12886 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 10 04:09:00 dist-test-slave-8w9v krb5kdc[12299](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334940, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.32.62@KRBTEST.COM
I20260810 04:09:00.364902 12895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:00.351757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47440 (local address 127.31.32.62:38215)
0810 04:09:00.352235 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0810 04:09:00.352245 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 04:09:00.352279 (+ 34us) server_negotiation.cc:407] Connection header received
0810 04:09:00.352467 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:09:00.352473 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:09:00.352542 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:09:00.354148 (+ 1606us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:09:00.354701 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:09:00.356523 (+ 1822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:09:00.357619 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:09:00.358185 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:09:00.358271 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:09:00.361020 (+ 2749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:09:00.361025 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:09:00.361025 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:09:00.361037 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:09:00.363019 (+ 1982us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:09:00.363412 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:09:00.363415 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:09:00.363415 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:09:00.363519 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:09:00.363693 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:09:00.363694 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:09:00.363694 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:09:00.363982 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:09:00.364134 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0810 04:09:00.364566 (+ 432us) server_negotiation.cc:299] Negotiation successful
0810 04:09:00.364683 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":432,"thread_start_us":89,"threads_started":1}
I20260810 04:09:00.393410 12886 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 04:09:00.393985 12886 catalog_manager.cc:1504] Loading token signing keys...
I20260810 04:09:00.402077 12886 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b615bc77cb4f4ad59550be465d809fb3: Generated new TSK 0
I20260810 04:09:00.402268 12886 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.32.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.32.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.32.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.1.keytab.
Entry for principal kudu/127.31.32.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.1.keytab.
I20260810 04:09:01.094627 31872 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskws89Io/build/release/bin/kudu
/tmp/dist-test-taskws89Io/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.32.20:53989/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.32.1:0
--local_ip_for_outbound_sockets=127.31.32.1
--webserver_interface=127.31.32.1
--webserver_port=0
--tserver_master_addrs=127.31.32.62:38215
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.1.keytab
--principal=kudu/127.31.32.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.32.20:33429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:09:01.183689 12924 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:09:01.183924 12924 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:09:01.183964 12924 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:09:01.185127 12924 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:09:01.185199 12924 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:09:01.185243 12924 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.32.1
I20260810 04:09:01.186713 12924 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.32.20:33429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.31.32.20:53989/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.1.keytab
--principal=kudu/127.31.32.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.32.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.32.1
--webserver_port=0
--tserver_master_addrs=127.31.32.62:38215
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12924
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.32.1
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:09:01.186975 12924 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:09:01.187206 12924 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:09:01.193197 12931 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:09:01.193225 12929 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:09:01.193645 12933 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:09:01.194476 12924 server_base.cc:1034] running on GCE node
I20260810 04:09:01.194680 12924 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:09:01.195047 12924 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:09:01.196195 12924 hybrid_clock.cc:648] HybridClock initialized: now 1786334941196160 us; error 47 us; skew 500 ppm
Aug 10 04:09:01 dist-test-slave-8w9v krb5kdc[12299](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334941, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:09:01.199435 12924 init.cc:377] Logged in from keytab as kudu/127.31.32.1@KRBTEST.COM (short username kudu)
I20260810 04:09:01.199944 12924 webserver.cc:460] Webserver started at http://127.31.32.1:37281/ using document root <none> and password file <none>
I20260810 04:09:01.200237 12924 fs_manager.cc:359] Metadata directory not provided
I20260810 04:09:01.200296 12924 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:09:01.200434 12924 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 04:09:01 dist-test-slave-8w9v krb5kdc[12299](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334941, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.1@KRBTEST.COM for HTTP/127.31.32.20@KRBTEST.COM
I20260810 04:09:01.214089 12924 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/minicluster-data/ts-0/data/instance:
uuid: "3fd89e4040e5434788438b468df8e19c"
format_stamp: "Formatted at 2026-08-10 04:09:01 on dist-test-slave-8w9v"
server_key: "39232c5b8c0a2314fae6660e581228dd"
server_key_iv: "fec1d0ad44bdcad8aebedcf7ce155fb1"
server_key_version: "kuduclusterkey@0"
I20260810 04:09:01.214640 12924 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/minicluster-data/ts-0/wal/instance:
uuid: "3fd89e4040e5434788438b468df8e19c"
format_stamp: "Formatted at 2026-08-10 04:09:01 on dist-test-slave-8w9v"
server_key: "39232c5b8c0a2314fae6660e581228dd"
server_key_iv: "fec1d0ad44bdcad8aebedcf7ce155fb1"
server_key_version: "kuduclusterkey@0"
I20260810 04:09:01.216607 12924 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.001s
I20260810 04:09:01.229899 12941 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:09:01.230223 12924 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260810 04:09:01.230355 12924 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/minicluster-data/ts-0/data,/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/minicluster-data/ts-0/wal
uuid: "3fd89e4040e5434788438b468df8e19c"
format_stamp: "Formatted at 2026-08-10 04:09:01 on dist-test-slave-8w9v"
server_key: "39232c5b8c0a2314fae6660e581228dd"
server_key_iv: "fec1d0ad44bdcad8aebedcf7ce155fb1"
server_key_version: "kuduclusterkey@0"
I20260810 04:09:01.230486 12924 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:09:01.260978 12924 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:09:01.263242 12924 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:09:01.263473 12924 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:09:01.263774 12924 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:09:01.264246 12924 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:09:01.264359 12924 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:09:01.264470 12924 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:09:01.264551 12924 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:09:01.271160 12924 rpc_server.cc:307] RPC server started. Bound to: 127.31.32.1:36929
I20260810 04:09:01.271672 12924 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/minicluster-data/ts-0/data/info.pb
I20260810 04:09:01.274469 13074 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.32.1:36929 every 8 connection(s)
I20260810 04:09:01.280794 31872 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskws89Io/build/release/bin/kudu as pid 12924
I20260810 04:09:01.280900 31872 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/minicluster-data/ts-0/wal/instance
I20260810 04:09:01.281083 31872 external_mini_cluster.cc:1428] Setting key 13090671a620093ed0cc4c24723802f7
Aug 10 04:09:01 dist-test-slave-8w9v krb5kdc[12299](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334941, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.1@KRBTEST.COM for kudu/127.31.32.62@KRBTEST.COM
I20260810 04:09:01.286950 13079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:01.275148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:38215 (local address 127.31.32.1:33771)
0810 04:09:01.275561 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0810 04:09:01.275604 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 04:09:01.276530 (+ 926us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:09:01.276808 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:09:01.276813 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:09:01.277011 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:09:01.277266 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:09:01.277271 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:09:01.278009 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:09:01.278010 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:09:01.279220 (+ 1210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:09:01.279223 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:09:01.279784 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:09:01.279785 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:09:01.279924 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:09:01.280581 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:09:01.280591 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:09:01.283050 (+ 2459us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:09:01.285179 (+ 2129us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:09:01.285187 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:09:01.285189 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:09:01.285512 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:09:01.285758 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:09:01.285759 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:09:01.285759 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:09:01.285900 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:09:01.286223 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:09:01.286228 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:09:01.286425 (+ 197us) client_negotiation.cc:769] Sending connection context
0810 04:09:01.286546 (+ 121us) client_negotiation.cc:240] Negotiation successful
0810 04:09:01.286708 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":96,"threads_started":1}
I20260810 04:09:01.287443 13080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:01.275174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.1:33771 (local address 127.31.32.62:38215)
0810 04:09:01.276427 (+ 1253us) server_negotiation.cc:206] Beginning negotiation
0810 04:09:01.276429 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:09:01.276441 (+ 12us) server_negotiation.cc:407] Connection header received
0810 04:09:01.276642 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:09:01.276643 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:09:01.276657 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:09:01.276736 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:09:01.277379 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:09:01.277899 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:09:01.279335 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:09:01.279643 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:09:01.279705 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:09:01.283158 (+ 3453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:09:01.283160 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:09:01.283161 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:09:01.283169 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:09:01.285081 (+ 1912us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:09:01.285610 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:09:01.285611 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:09:01.285612 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:09:01.285683 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:09:01.285984 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:09:01.285985 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:09:01.285985 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:09:01.286128 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:09:01.286194 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0810 04:09:01.287241 (+ 1047us) server_negotiation.cc:299] Negotiation successful
0810 04:09:01.287345 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1222,"thread_start_us":83,"threads_started":1}
I20260810 04:09:01.287774 13076 heartbeater.cc:347] Connected to a master server at 127.31.32.62:38215
I20260810 04:09:01.287868 13076 heartbeater.cc:464] Registering TS with master...
I20260810 04:09:01.288060 13076 heartbeater.cc:511] Master 127.31.32.62:38215 requested a full tablet report, sending...
I20260810 04:09:01.288650 12831 ts_manager.cc:194] Registered new tserver with Master: 3fd89e4040e5434788438b468df8e19c (127.31.32.1:36929)
I20260810 04:09:01.289589 12831 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.32.1@KRBTEST.COM'} at 127.31.32.1:33771
WARNING: no policy specified for kudu/127.31.32.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.32.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.32.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.2.keytab.
Entry for principal kudu/127.31.32.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.2.keytab.
I20260810 04:09:01.329275 31872 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskws89Io/build/release/bin/kudu
/tmp/dist-test-taskws89Io/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.32.20:53989/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.32.2:0
--local_ip_for_outbound_sockets=127.31.32.2
--webserver_interface=127.31.32.2
--webserver_port=0
--tserver_master_addrs=127.31.32.62:38215
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.2.keytab
--principal=kudu/127.31.32.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.32.20:33429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:09:01.451161 13083 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:09:01.451512 13083 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:09:01.451602 13083 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:09:01.453325 13083 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:09:01.453557 13083 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:09:01.453646 13083 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.32.2
I20260810 04:09:01.455938 13083 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.32.20:33429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.31.32.20:53989/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.2.keytab
--principal=kudu/127.31.32.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.32.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.32.2
--webserver_port=0
--tserver_master_addrs=127.31.32.62:38215
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13083
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.32.2
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:09:01.456468 13083 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:09:01.456802 13083 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:09:01.463650 13093 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
Aug 10 04:09:01 dist-test-slave-8w9v krb5kdc[12299](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786334941, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260810 04:09:01.463650 13091 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:09:01.464624 13090 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:09:01.464924 13083 server_base.cc:1034] running on GCE node
I20260810 04:09:01.465127 13083 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:09:01.465392 13083 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:09:01.466555 13083 hybrid_clock.cc:648] HybridClock initialized: now 1786334941466535 us; error 31 us; skew 500 ppm
Aug 10 04:09:01 dist-test-slave-8w9v krb5kdc[12299](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334941, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:09:01.469935 13083 init.cc:377] Logged in from keytab as kudu/127.31.32.2@KRBTEST.COM (short username kudu)
I20260810 04:09:01.470431 13083 webserver.cc:460] Webserver started at http://127.31.32.2:37735/ using document root <none> and password file <none>
I20260810 04:09:01.470707 13083 fs_manager.cc:359] Metadata directory not provided
I20260810 04:09:01.470762 13083 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:09:01.470899 13083 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 04:09:01 dist-test-slave-8w9v krb5kdc[12299](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334941, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.2@KRBTEST.COM for HTTP/127.31.32.20@KRBTEST.COM
I20260810 04:09:01.491515 13083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/minicluster-data/ts-1/data/instance:
uuid: "89e088f695d34b37997b7449193be12a"
format_stamp: "Formatted at 2026-08-10 04:09:01 on dist-test-slave-8w9v"
server_key: "8ddb345c61d391b2b0ae891e8cee809d"
server_key_iv: "9ae03b6db1b4b41c45fc1f5159ba9b4e"
server_key_version: "kuduclusterkey@0"
I20260810 04:09:01.493160 13083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/minicluster-data/ts-1/wal/instance:
uuid: "89e088f695d34b37997b7449193be12a"
format_stamp: "Formatted at 2026-08-10 04:09:01 on dist-test-slave-8w9v"
server_key: "8ddb345c61d391b2b0ae891e8cee809d"
server_key_iv: "9ae03b6db1b4b41c45fc1f5159ba9b4e"
server_key_version: "kuduclusterkey@0"
I20260810 04:09:01.497080 13083 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.000s
I20260810 04:09:01.517786 13102 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:09:01.518065 13083 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:09:01.518229 13083 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/minicluster-data/ts-1/data,/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/minicluster-data/ts-1/wal
uuid: "89e088f695d34b37997b7449193be12a"
format_stamp: "Formatted at 2026-08-10 04:09:01 on dist-test-slave-8w9v"
server_key: "8ddb345c61d391b2b0ae891e8cee809d"
server_key_iv: "9ae03b6db1b4b41c45fc1f5159ba9b4e"
server_key_version: "kuduclusterkey@0"
I20260810 04:09:01.518409 13083 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:09:01.538743 13083 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:09:01.541055 13083 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:09:01.541222 13083 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:09:01.541455 13083 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:09:01.541843 13083 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:09:01.541884 13083 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:09:01.541914 13083 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:09:01.541965 13083 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:09:01.548974 13083 rpc_server.cc:307] RPC server started. Bound to: 127.31.32.2:42877
I20260810 04:09:01.549528 13083 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/minicluster-data/ts-1/data/info.pb
I20260810 04:09:01.551151 13231 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.32.2:42877 every 8 connection(s)
I20260810 04:09:01.553341 31872 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskws89Io/build/release/bin/kudu as pid 13083
I20260810 04:09:01.553457 31872 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/minicluster-data/ts-1/wal/instance
I20260810 04:09:01.553639 31872 external_mini_cluster.cc:1428] Setting key a7f11e764bf9bb989a84a334a6c4aab7
Aug 10 04:09:01 dist-test-slave-8w9v krb5kdc[12299](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334941, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.2@KRBTEST.COM for kudu/127.31.32.62@KRBTEST.COM
I20260810 04:09:01.563526 13080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:01.552166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.2:54673 (local address 127.31.32.62:38215)
0810 04:09:01.552218 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0810 04:09:01.552221 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:09:01.552543 (+ 322us) server_negotiation.cc:407] Connection header received
0810 04:09:01.553259 (+ 716us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:09:01.553261 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:09:01.553273 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:09:01.553304 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:09:01.554768 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:09:01.555533 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:09:01.557039 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:09:01.557368 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:09:01.557414 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:09:01.560188 (+ 2774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:09:01.560190 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:09:01.560190 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:09:01.560198 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:09:01.561890 (+ 1692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:09:01.562393 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:09:01.562395 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:09:01.562395 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:09:01.562456 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:09:01.562738 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:09:01.562739 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:09:01.562740 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:09:01.562903 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:09:01.562957 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0810 04:09:01.563335 (+ 378us) server_negotiation.cc:299] Negotiation successful
0810 04:09:01.563428 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260810 04:09:01.564296 13235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:01.551850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:38215 (local address 127.31.32.2:54673)
0810 04:09:01.552414 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0810 04:09:01.552459 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 04:09:01.553134 (+ 675us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:09:01.553505 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:09:01.553509 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:09:01.553738 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:09:01.554654 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:09:01.554662 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:09:01.555654 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:09:01.555655 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:09:01.556935 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:09:01.556938 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:09:01.557524 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:09:01.557525 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:09:01.557659 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:09:01.558236 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:09:01.558256 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:09:01.560047 (+ 1791us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:09:01.561986 (+ 1939us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:09:01.561993 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:09:01.561995 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:09:01.562294 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:09:01.562527 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:09:01.562528 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:09:01.562528 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:09:01.562659 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:09:01.562991 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:09:01.562998 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:09:01.563167 (+ 169us) client_negotiation.cc:769] Sending connection context
0810 04:09:01.563864 (+ 697us) client_negotiation.cc:240] Negotiation successful
0810 04:09:01.564026 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":84,"threads_started":1}
I20260810 04:09:01.564953 13232 heartbeater.cc:347] Connected to a master server at 127.31.32.62:38215
I20260810 04:09:01.565047 13232 heartbeater.cc:464] Registering TS with master...
I20260810 04:09:01.565228 13232 heartbeater.cc:511] Master 127.31.32.62:38215 requested a full tablet report, sending...
I20260810 04:09:01.565683 12831 ts_manager.cc:194] Registered new tserver with Master: 89e088f695d34b37997b7449193be12a (127.31.32.2:42877)
I20260810 04:09:01.566159 12831 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.32.2@KRBTEST.COM'} at 127.31.32.2:54673
WARNING: no policy specified for kudu/127.31.32.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.32.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.32.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.3.keytab.
Entry for principal kudu/127.31.32.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.3.keytab.
I20260810 04:09:01.600057 31872 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskws89Io/build/release/bin/kudu
/tmp/dist-test-taskws89Io/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.32.20:53989/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.32.3:0
--local_ip_for_outbound_sockets=127.31.32.3
--webserver_interface=127.31.32.3
--webserver_port=0
--tserver_master_addrs=127.31.32.62:38215
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.3.keytab
--principal=kudu/127.31.32.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.32.20:33429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:09:01.725857 13239 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:09:01.726130 13239 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:09:01.726171 13239 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:09:01.727857 13239 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:09:01.727947 13239 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:09:01.728012 13239 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.32.3
I20260810 04:09:01.730158 13239 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.32.20:33429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.31.32.20:53989/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.3.keytab
--principal=kudu/127.31.32.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.32.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.32.3
--webserver_port=0
--tserver_master_addrs=127.31.32.62:38215
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13239
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.32.3
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:09:01.730558 13239 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:09:01.730873 13239 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:09:01.737462 13248 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:09:01.737707 13246 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:09:01.738770 13239 server_base.cc:1034] running on GCE node
W20260810 04:09:01.740278 13244 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:09:01.740595 13239 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:09:01.740880 13239 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:09:01.745491 13239 hybrid_clock.cc:648] HybridClock initialized: now 1786334941745462 us; error 39 us; skew 500 ppm
Aug 10 04:09:01 dist-test-slave-8w9v krb5kdc[12299](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334941, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:09:01.750469 13239 init.cc:377] Logged in from keytab as kudu/127.31.32.3@KRBTEST.COM (short username kudu)
I20260810 04:09:01.751358 13239 webserver.cc:460] Webserver started at http://127.31.32.3:37617/ using document root <none> and password file <none>
I20260810 04:09:01.752002 13239 fs_manager.cc:359] Metadata directory not provided
I20260810 04:09:01.752192 13239 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:09:01.752528 13239 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 04:09:01 dist-test-slave-8w9v krb5kdc[12299](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334941, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.3@KRBTEST.COM for HTTP/127.31.32.20@KRBTEST.COM
I20260810 04:09:01.772769 13239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/minicluster-data/ts-2/data/instance:
uuid: "fa3ec6a7a2584d78a3a51ebf25bee9c0"
format_stamp: "Formatted at 2026-08-10 04:09:01 on dist-test-slave-8w9v"
server_key: "71f1bce9141c11a3a7907be824f07ae2"
server_key_iv: "aad8a59951cfb6f6eadb9b2c61d37556"
server_key_version: "kuduclusterkey@0"
I20260810 04:09:01.773209 13239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/minicluster-data/ts-2/wal/instance:
uuid: "fa3ec6a7a2584d78a3a51ebf25bee9c0"
format_stamp: "Formatted at 2026-08-10 04:09:01 on dist-test-slave-8w9v"
server_key: "71f1bce9141c11a3a7907be824f07ae2"
server_key_iv: "aad8a59951cfb6f6eadb9b2c61d37556"
server_key_version: "kuduclusterkey@0"
I20260810 04:09:01.775030 13239 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260810 04:09:01.791666 13255 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:09:01.791986 13239 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.000s sys 0.001s
I20260810 04:09:01.792140 13239 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/minicluster-data/ts-2/data,/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/minicluster-data/ts-2/wal
uuid: "fa3ec6a7a2584d78a3a51ebf25bee9c0"
format_stamp: "Formatted at 2026-08-10 04:09:01 on dist-test-slave-8w9v"
server_key: "71f1bce9141c11a3a7907be824f07ae2"
server_key_iv: "aad8a59951cfb6f6eadb9b2c61d37556"
server_key_version: "kuduclusterkey@0"
I20260810 04:09:01.792253 13239 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:09:01.810015 13239 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:09:01.812408 13239 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:09:01.812547 13239 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:09:01.812774 13239 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:09:01.813117 13239 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:09:01.813177 13239 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:09:01.813215 13239 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:09:01.813241 13239 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:09:01.821960 13239 rpc_server.cc:307] RPC server started. Bound to: 127.31.32.3:32797
I20260810 04:09:01.822389 13239 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/minicluster-data/ts-2/data/info.pb
I20260810 04:09:01.824007 13378 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.32.3:32797 every 8 connection(s)
I20260810 04:09:01.825042 31872 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskws89Io/build/release/bin/kudu as pid 13239
I20260810 04:09:01.825146 31872 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/minicluster-data/ts-2/wal/instance
I20260810 04:09:01.825361 31872 external_mini_cluster.cc:1428] Setting key 5bdb96c33e363b898dba51c20eda50c8
Aug 10 04:09:01 dist-test-slave-8w9v krb5kdc[12299](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334941, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.3@KRBTEST.COM for kudu/127.31.32.62@KRBTEST.COM
I20260810 04:09:01.833425 13080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:01.823902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.3:57759 (local address 127.31.32.62:38215)
0810 04:09:01.823961 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0810 04:09:01.823964 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:09:01.824378 (+ 414us) server_negotiation.cc:407] Connection header received
0810 04:09:01.825068 (+ 690us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:09:01.825070 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:09:01.825082 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:09:01.825112 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:09:01.825925 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:09:01.826660 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:09:01.827776 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:09:01.828004 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:09:01.828085 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:09:01.830465 (+ 2380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:09:01.830467 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:09:01.830467 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:09:01.830475 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:09:01.832021 (+ 1546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:09:01.832469 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:09:01.832470 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:09:01.832470 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:09:01.832527 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:09:01.832785 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:09:01.832786 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:09:01.832786 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:09:01.832906 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:09:01.832975 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0810 04:09:01.833259 (+ 284us) server_negotiation.cc:299] Negotiation successful
0810 04:09:01.833337 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260810 04:09:01.833791 13381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:01.823555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:38215 (local address 127.31.32.3:57759)
0810 04:09:01.824245 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0810 04:09:01.824283 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0810 04:09:01.824947 (+ 664us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:09:01.825287 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:09:01.825291 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:09:01.825535 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:09:01.825817 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:09:01.825824 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:09:01.826793 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:09:01.826794 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:09:01.827671 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:09:01.827673 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:09:01.828103 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:09:01.828104 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:09:01.828263 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:09:01.828801 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:09:01.828810 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:09:01.830386 (+ 1576us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:09:01.832157 (+ 1771us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:09:01.832163 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:09:01.832165 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:09:01.832391 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:09:01.832600 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:09:01.832601 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:09:01.832601 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:09:01.832714 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:09:01.832979 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:09:01.832986 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:09:01.833127 (+ 141us) client_negotiation.cc:769] Sending connection context
0810 04:09:01.833225 (+ 98us) client_negotiation.cc:240] Negotiation successful
0810 04:09:01.833369 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":651,"thread_start_us":147,"threads_started":1}
I20260810 04:09:01.834381 13379 heartbeater.cc:347] Connected to a master server at 127.31.32.62:38215
I20260810 04:09:01.834491 13379 heartbeater.cc:464] Registering TS with master...
I20260810 04:09:01.834633 13379 heartbeater.cc:511] Master 127.31.32.62:38215 requested a full tablet report, sending...
I20260810 04:09:01.834973 12831 ts_manager.cc:194] Registered new tserver with Master: fa3ec6a7a2584d78a3a51ebf25bee9c0 (127.31.32.3:32797)
I20260810 04:09:01.835402 12831 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.32.3@KRBTEST.COM'} at 127.31.32.3:57759
I20260810 04:09:01.836838 31872 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 04:09:01.837522 31872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskws89Io/build/release/bin/kudu with pid 12924
I20260810 04:09:01.847185 31872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskws89Io/build/release/bin/kudu with pid 13083
I20260810 04:09:01.855975 31872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskws89Io/build/release/bin/kudu with pid 13239
I20260810 04:09:01.862452 31872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskws89Io/build/release/bin/kudu with pid 12793
I20260810 04:09:01.870814 31872 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskws89Io/build/release/bin/kudu
/tmp/dist-test-taskws89Io/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.32.62:38215
--webserver_interface=127.31.32.62
--webserver_port=38119
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.62.keytab
--principal=kudu/127.31.32.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.32.20:33429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.32.62:38215
--ranger_config_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.32.20:53989/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.31.32.20:53989/kms with env {KRB5CCNAME=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:09:01.951836 13382 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:09:01.952028 13382 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:09:01.952050 13382 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:09:01.953122 13382 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:09:01.953167 13382 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:09:01.953182 13382 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:09:01.953195 13382 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:09:01.953208 13382 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:09:01.954577 13382 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.32.20:33429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.31.32.20:53989/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.31.32.62:38215
--ranger_config_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.62.keytab
--principal=kudu/127.31.32.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.32.62:38215
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.32.62
--webserver_port=38119
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13382
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:09:01.954746 13382 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:09:01.954900 13382 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:09:01.959826 13389 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:09:01.959882 13382 server_base.cc:1034] running on GCE node
W20260810 04:09:01.959844 13388 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:09:01.959892 13392 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:09:01.960300 13382 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:09:01.960546 13382 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:09:01.961678 13382 hybrid_clock.cc:648] HybridClock initialized: now 1786334941961666 us; error 31 us; skew 500 ppm
Aug 10 04:09:01 dist-test-slave-8w9v krb5kdc[12299](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334941, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:09:01.964190 13382 init.cc:377] Logged in from keytab as kudu/127.31.32.62@KRBTEST.COM (short username kudu)
I20260810 04:09:01.964578 13382 webserver.cc:460] Webserver started at http://127.31.32.62:38119/ using document root <none> and password file <none>
I20260810 04:09:01.964785 13382 fs_manager.cc:359] Metadata directory not provided
I20260810 04:09:01.964838 13382 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 10 04:09:01 dist-test-slave-8w9v krb5kdc[12299](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334941, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.62@KRBTEST.COM for HTTP/127.31.32.20@KRBTEST.COM
I20260810 04:09:02.011879 13382 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:09:02.013202 13400 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:09:02.013402 13382 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260810 04:09:02.013484 13382 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/minicluster-data/master-0/data,/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/minicluster-data/master-0/wal
uuid: "b615bc77cb4f4ad59550be465d809fb3"
format_stamp: "Formatted at 2026-08-10 04:09:00 on dist-test-slave-8w9v"
server_key: "79a01bdace635be45311ba7ac323fef6"
server_key_iv: "aeca28dbf5cd96d77752a6fe767b02e5"
server_key_version: "kuduclusterkey@0"
I20260810 04:09:02.013962 13382 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:09:02.031746 13382 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:09:02.034000 13382 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:09:02.034162 13382 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:09:02.039422 13382 rpc_server.cc:307] RPC server started. Bound to: 127.31.32.62:38215
I20260810 04:09:02.039450 13457 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.32.62:38215 every 8 connection(s)
I20260810 04:09:02.039906 13382 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/minicluster-data/master-0/data/info.pb
I20260810 04:09:02.041249 13458 sys_catalog.cc:263] Verifying existing consensus state
I20260810 04:09:02.041924 13458 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b615bc77cb4f4ad59550be465d809fb3: Bootstrap starting.
I20260810 04:09:02.046829 31872 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskws89Io/build/release/bin/kudu as pid 13382
I20260810 04:09:02.047140 31872 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskws89Io/build/release/bin/kudu
/tmp/dist-test-taskws89Io/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.32.20:53989/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.32.1:36929
--local_ip_for_outbound_sockets=127.31.32.1
--tserver_master_addrs=127.31.32.62:38215
--webserver_port=37281
--webserver_interface=127.31.32.1
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.1.keytab
--principal=kudu/127.31.32.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.32.20:33429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.31.32.20:53989/kms with env {KRB5CCNAME=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 04:09:02.050364 13458 log.cc:824] T 00000000000000000000000000000000 P b615bc77cb4f4ad59550be465d809fb3: Log is configured to *not* fsync() on all Append() calls
I20260810 04:09:02.052089 13458 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b615bc77cb4f4ad59550be465d809fb3: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 04:09:02.052403 13458 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b615bc77cb4f4ad59550be465d809fb3: Bootstrap complete.
I20260810 04:09:02.054714 13458 raft_consensus.cc:348] T 00000000000000000000000000000000 P b615bc77cb4f4ad59550be465d809fb3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b615bc77cb4f4ad59550be465d809fb3" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 38215 } }
I20260810 04:09:02.054960 13458 raft_consensus.cc:729] T 00000000000000000000000000000000 P b615bc77cb4f4ad59550be465d809fb3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b615bc77cb4f4ad59550be465d809fb3, State: Initialized, Role: FOLLOWER
I20260810 04:09:02.055120 13458 consensus_queue.cc:260] T 00000000000000000000000000000000 P b615bc77cb4f4ad59550be465d809fb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b615bc77cb4f4ad59550be465d809fb3" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 38215 } }
I20260810 04:09:02.055219 13458 raft_consensus.cc:388] T 00000000000000000000000000000000 P b615bc77cb4f4ad59550be465d809fb3 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 04:09:02.055285 13458 raft_consensus.cc:482] T 00000000000000000000000000000000 P b615bc77cb4f4ad59550be465d809fb3 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 04:09:02.055325 13458 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b615bc77cb4f4ad59550be465d809fb3 [term 1 FOLLOWER]: Advancing to term 2
I20260810 04:09:02.056420 13458 raft_consensus.cc:504] T 00000000000000000000000000000000 P b615bc77cb4f4ad59550be465d809fb3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b615bc77cb4f4ad59550be465d809fb3" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 38215 } }
I20260810 04:09:02.056542 13458 leader_election.cc:302] T 00000000000000000000000000000000 P b615bc77cb4f4ad59550be465d809fb3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b615bc77cb4f4ad59550be465d809fb3; no voters:
I20260810 04:09:02.056735 13458 leader_election.cc:288] T 00000000000000000000000000000000 P b615bc77cb4f4ad59550be465d809fb3 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260810 04:09:02.056839 13462 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b615bc77cb4f4ad59550be465d809fb3 [term 2 FOLLOWER]: Leader election won for term 2
I20260810 04:09:02.057001 13462 raft_consensus.cc:686] T 00000000000000000000000000000000 P b615bc77cb4f4ad59550be465d809fb3 [term 2 LEADER]: Becoming Leader. State: Replica: b615bc77cb4f4ad59550be465d809fb3, State: Running, Role: LEADER
I20260810 04:09:02.057020 13458 sys_catalog.cc:564] T 00000000000000000000000000000000 P b615bc77cb4f4ad59550be465d809fb3 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:09:02.057209 13462 consensus_queue.cc:237] T 00000000000000000000000000000000 P b615bc77cb4f4ad59550be465d809fb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b615bc77cb4f4ad59550be465d809fb3" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 38215 } }
I20260810 04:09:02.057549 13462 sys_catalog.cc:455] T 00000000000000000000000000000000 P b615bc77cb4f4ad59550be465d809fb3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "b615bc77cb4f4ad59550be465d809fb3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b615bc77cb4f4ad59550be465d809fb3" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 38215 } } }
I20260810 04:09:02.057701 13462 sys_catalog.cc:458] T 00000000000000000000000000000000 P b615bc77cb4f4ad59550be465d809fb3 [sys.catalog]: This master's current role is: LEADER
I20260810 04:09:02.057883 13462 sys_catalog.cc:455] T 00000000000000000000000000000000 P b615bc77cb4f4ad59550be465d809fb3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b615bc77cb4f4ad59550be465d809fb3. Latest consensus state: current_term: 2 leader_uuid: "b615bc77cb4f4ad59550be465d809fb3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b615bc77cb4f4ad59550be465d809fb3" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 38215 } } }
I20260810 04:09:02.057983 13462 sys_catalog.cc:458] T 00000000000000000000000000000000 P b615bc77cb4f4ad59550be465d809fb3 [sys.catalog]: This master's current role is: LEADER
I20260810 04:09:02.058163 13471 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 04:09:02.058598 13458 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260810 04:09:02.058801 13471 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 04:09:02.058991 13471 catalog_manager.cc:1259] Loaded cluster ID: 0dcf7608dc5a4b94a479037fd605a265
I20260810 04:09:02.059069 13471 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 04:09:02.063563 13471 catalog_manager.cc:1504] Loading token signing keys...
I20260810 04:09:02.063830 13471 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b615bc77cb4f4ad59550be465d809fb3: Loaded TSK: 0
I20260810 04:09:02.064047 13471 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 04:09:02.139693 13460 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:09:02.140086 13460 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:09:02.140228 13460 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:09:02.141942 13460 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:09:02.142055 13460 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:09:02.142148 13460 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.32.1
I20260810 04:09:02.145726 13460 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.32.20:33429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.31.32.20:53989/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.1.keytab
--principal=kudu/127.31.32.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.32.1:36929
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.32.1
--webserver_port=37281
--tserver_master_addrs=127.31.32.62:38215
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13460
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.32.1
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:09:02.145996 13460 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:09:02.146344 13460 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:09:02.153033 13500 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:09:02.153097 13503 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:09:02.153581 13499 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:09:02.153571 13460 server_base.cc:1034] running on GCE node
I20260810 04:09:02.154001 13460 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:09:02.154321 13460 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:09:02.157608 13460 hybrid_clock.cc:648] HybridClock initialized: now 1786334942157583 us; error 34 us; skew 500 ppm
Aug 10 04:09:02 dist-test-slave-8w9v krb5kdc[12299](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334942, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:09:02.161099 13460 init.cc:377] Logged in from keytab as kudu/127.31.32.1@KRBTEST.COM (short username kudu)
I20260810 04:09:02.161614 13460 webserver.cc:460] Webserver started at http://127.31.32.1:37281/ using document root <none> and password file <none>
I20260810 04:09:02.161876 13460 fs_manager.cc:359] Metadata directory not provided
I20260810 04:09:02.161954 13460 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 10 04:09:02 dist-test-slave-8w9v krb5kdc[12299](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334942, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.1@KRBTEST.COM for HTTP/127.31.32.20@KRBTEST.COM
I20260810 04:09:02.188560 13460 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260810 04:09:02.189508 13512 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:09:02.189718 13460 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260810 04:09:02.189831 13460 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/minicluster-data/ts-0/data,/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/minicluster-data/ts-0/wal
uuid: "3fd89e4040e5434788438b468df8e19c"
format_stamp: "Formatted at 2026-08-10 04:09:01 on dist-test-slave-8w9v"
server_key: "39232c5b8c0a2314fae6660e581228dd"
server_key_iv: "fec1d0ad44bdcad8aebedcf7ce155fb1"
server_key_version: "kuduclusterkey@0"
I20260810 04:09:02.190268 13460 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:09:02.206187 13460 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:09:02.208590 13460 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:09:02.208773 13460 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:09:02.209030 13460 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:09:02.209430 13460 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:09:02.209476 13460 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:09:02.209514 13460 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:09:02.209540 13460 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:09:02.216706 13460 rpc_server.cc:307] RPC server started. Bound to: 127.31.32.1:36929
I20260810 04:09:02.217192 13460 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/minicluster-data/ts-0/data/info.pb
I20260810 04:09:02.224860 13634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.32.1:36929 every 8 connection(s)
I20260810 04:09:02.226087 31872 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskws89Io/build/release/bin/kudu as pid 13460
I20260810 04:09:02.226414 31872 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskws89Io/build/release/bin/kudu
/tmp/dist-test-taskws89Io/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.32.20:53989/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.32.2:42877
--local_ip_for_outbound_sockets=127.31.32.2
--tserver_master_addrs=127.31.32.62:38215
--webserver_port=37735
--webserver_interface=127.31.32.2
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.2.keytab
--principal=kudu/127.31.32.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.32.20:33429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.31.32.20:53989/kms with env {KRB5CCNAME=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 04:09:02 dist-test-slave-8w9v krb5kdc[12299](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334942, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.1@KRBTEST.COM for kudu/127.31.32.62@KRBTEST.COM
I20260810 04:09:02.248386 13638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:02.228824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:38215 (local address 127.31.32.1:55823)
0810 04:09:02.232238 (+ 3414us) negotiation.cc:107] Waiting for socket to connect
0810 04:09:02.232293 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0810 04:09:02.232890 (+ 597us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:09:02.233584 (+ 694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:09:02.233588 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:09:02.233854 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:09:02.234154 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:09:02.234161 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:09:02.235946 (+ 1785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:09:02.235947 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:09:02.237288 (+ 1341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:09:02.237291 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:09:02.238167 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:09:02.238168 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:09:02.238327 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:09:02.238971 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:09:02.238982 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:09:02.240687 (+ 1705us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:09:02.246165 (+ 5478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:09:02.246173 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:09:02.246175 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:09:02.246490 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:09:02.246819 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:09:02.246820 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:09:02.246820 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:09:02.246974 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:09:02.247534 (+ 560us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:09:02.247541 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:09:02.247759 (+ 218us) client_negotiation.cc:769] Sending connection context
0810 04:09:02.247878 (+ 119us) client_negotiation.cc:240] Negotiation successful
0810 04:09:02.248077 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3366,"thread_start_us":104,"threads_started":1}
I20260810 04:09:02.249280 13639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:02.229232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.1:55823 (local address 127.31.32.62:38215)
0810 04:09:02.233093 (+ 3861us) server_negotiation.cc:206] Beginning negotiation
0810 04:09:02.233101 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 04:09:02.233133 (+ 32us) server_negotiation.cc:407] Connection header received
0810 04:09:02.233302 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:09:02.233308 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:09:02.233369 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:09:02.234307 (+ 938us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:09:02.234396 (+ 89us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:09:02.235816 (+ 1420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:09:02.237406 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:09:02.238043 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:09:02.239737 (+ 1694us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:09:02.240834 (+ 1097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:09:02.240838 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:09:02.240839 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:09:02.240850 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:09:02.246023 (+ 5173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:09:02.246592 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:09:02.246595 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:09:02.246595 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:09:02.246702 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:09:02.247074 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:09:02.247075 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:09:02.247075 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:09:02.247410 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:09:02.248872 (+ 1462us) server_negotiation.cc:1035] Waiting for connection context
0810 04:09:02.248949 (+ 77us) server_negotiation.cc:299] Negotiation successful
0810 04:09:02.249076 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3806,"thread_start_us":103,"threads_started":1}
I20260810 04:09:02.250200 13635 heartbeater.cc:347] Connected to a master server at 127.31.32.62:38215
I20260810 04:09:02.250370 13635 heartbeater.cc:464] Registering TS with master...
I20260810 04:09:02.250589 13635 heartbeater.cc:511] Master 127.31.32.62:38215 requested a full tablet report, sending...
W20260810 04:09:02.251538 13635 heartbeater.cc:644] Failed to heartbeat to 127.31.32.62:38215 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260810 04:09:02.251633 13635 heartbeater.cc:464] Registering TS with master...
I20260810 04:09:02.251711 13635 heartbeater.cc:511] Master 127.31.32.62:38215 requested a full tablet report, sending...
I20260810 04:09:02.252722 13635 heartbeater.cc:464] Registering TS with master...
I20260810 04:09:02.252822 13635 heartbeater.cc:511] Master 127.31.32.62:38215 requested a full tablet report, sending...
W20260810 04:09:02.254045 13635 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 04:09:02.333026 13637 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:09:02.333281 13637 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:09:02.333314 13637 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:09:02.334944 13637 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:09:02.335001 13637 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:09:02.335040 13637 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.32.2
I20260810 04:09:02.337087 13637 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.32.20:33429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.31.32.20:53989/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.2.keytab
--principal=kudu/127.31.32.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.32.2:42877
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.32.2
--webserver_port=37735
--tserver_master_addrs=127.31.32.62:38215
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13637
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.32.2
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:09:02.337352 13637 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:09:02.337594 13637 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:09:02.342656 13646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:09:02.342974 13645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:09:02.343185 13648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:09:02.343230 13637 server_base.cc:1034] running on GCE node
I20260810 04:09:02.343461 13637 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:09:02.343744 13637 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:09:02.344934 13637 hybrid_clock.cc:648] HybridClock initialized: now 1786334942344913 us; error 36 us; skew 500 ppm
Aug 10 04:09:02 dist-test-slave-8w9v krb5kdc[12299](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334942, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:09:02.348323 13637 init.cc:377] Logged in from keytab as kudu/127.31.32.2@KRBTEST.COM (short username kudu)
I20260810 04:09:02.348915 13637 webserver.cc:460] Webserver started at http://127.31.32.2:37735/ using document root <none> and password file <none>
I20260810 04:09:02.349292 13637 fs_manager.cc:359] Metadata directory not provided
I20260810 04:09:02.349471 13637 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 10 04:09:02 dist-test-slave-8w9v krb5kdc[12299](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334942, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.2@KRBTEST.COM for HTTP/127.31.32.20@KRBTEST.COM
I20260810 04:09:02.363462 13637 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:09:02.364149 13657 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:09:02.364382 13637 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:09:02.364518 13637 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/minicluster-data/ts-1/data,/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/minicluster-data/ts-1/wal
uuid: "89e088f695d34b37997b7449193be12a"
format_stamp: "Formatted at 2026-08-10 04:09:01 on dist-test-slave-8w9v"
server_key: "8ddb345c61d391b2b0ae891e8cee809d"
server_key_iv: "9ae03b6db1b4b41c45fc1f5159ba9b4e"
server_key_version: "kuduclusterkey@0"
I20260810 04:09:02.365028 13637 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:09:02.386328 13637 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:09:02.388969 13637 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:09:02.389130 13637 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:09:02.389418 13637 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:09:02.389801 13637 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:09:02.389849 13637 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:09:02.389881 13637 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:09:02.389905 13637 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:09:02.397697 13637 rpc_server.cc:307] RPC server started. Bound to: 127.31.32.2:42877
I20260810 04:09:02.397785 13783 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.32.2:42877 every 8 connection(s)
I20260810 04:09:02.398178 13637 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/minicluster-data/ts-1/data/info.pb
I20260810 04:09:02.404181 31872 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskws89Io/build/release/bin/kudu as pid 13637
I20260810 04:09:02.404524 31872 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskws89Io/build/release/bin/kudu
/tmp/dist-test-taskws89Io/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.32.20:53989/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.32.3:32797
--local_ip_for_outbound_sockets=127.31.32.3
--tserver_master_addrs=127.31.32.62:38215
--webserver_port=37617
--webserver_interface=127.31.32.3
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.3.keytab
--principal=kudu/127.31.32.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.32.20:33429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.31.32.20:53989/kms with env {KRB5CCNAME=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 04:09:02 dist-test-slave-8w9v krb5kdc[12299](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334942, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.2@KRBTEST.COM for kudu/127.31.32.62@KRBTEST.COM
I20260810 04:09:02.417956 13639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:02.399049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.2:56993 (local address 127.31.32.62:38215)
0810 04:09:02.399114 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0810 04:09:02.399117 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:09:02.399353 (+ 236us) server_negotiation.cc:407] Connection header received
0810 04:09:02.400173 (+ 820us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:09:02.400175 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:09:02.400195 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:09:02.400278 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:09:02.401060 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:09:02.401867 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:09:02.402935 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:09:02.403258 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:09:02.403346 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:09:02.406413 (+ 3067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:09:02.406414 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:09:02.406415 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:09:02.406422 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:09:02.408709 (+ 2287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:09:02.416717 (+ 8008us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:09:02.416719 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:09:02.416719 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:09:02.416799 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:09:02.417131 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:09:02.417132 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:09:02.417132 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:09:02.417293 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:09:02.417342 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0810 04:09:02.417755 (+ 413us) server_negotiation.cc:299] Negotiation successful
0810 04:09:02.417856 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260810 04:09:02.418907 13786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:02.399019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:38215 (local address 127.31.32.2:56993)
0810 04:09:02.399207 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0810 04:09:02.399248 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 04:09:02.399949 (+ 701us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:09:02.400352 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:09:02.400357 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:09:02.400625 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:09:02.400940 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:09:02.400947 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:09:02.401963 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:09:02.401965 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:09:02.402842 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:09:02.402845 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:09:02.403383 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:09:02.403384 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:09:02.403488 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:09:02.404080 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:09:02.404091 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:09:02.406098 (+ 2007us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:09:02.408838 (+ 2740us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:09:02.408845 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:09:02.408848 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:09:02.409751 (+ 903us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:09:02.416886 (+ 7135us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:09:02.416887 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:09:02.416887 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:09:02.417043 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:09:02.417395 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:09:02.417402 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:09:02.417588 (+ 186us) client_negotiation.cc:769] Sending connection context
0810 04:09:02.418492 (+ 904us) client_negotiation.cc:240] Negotiation successful
0810 04:09:02.418674 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":98,"threads_started":1}
I20260810 04:09:02.419700 13784 heartbeater.cc:347] Connected to a master server at 127.31.32.62:38215
I20260810 04:09:02.419826 13784 heartbeater.cc:464] Registering TS with master...
I20260810 04:09:02.420037 13784 heartbeater.cc:511] Master 127.31.32.62:38215 requested a full tablet report, sending...
W20260810 04:09:02.420837 13784 heartbeater.cc:644] Failed to heartbeat to 127.31.32.62:38215 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260810 04:09:02.420921 13784 heartbeater.cc:464] Registering TS with master...
I20260810 04:09:02.420996 13784 heartbeater.cc:511] Master 127.31.32.62:38215 requested a full tablet report, sending...
I20260810 04:09:02.421348 13784 heartbeater.cc:464] Registering TS with master...
I20260810 04:09:02.421435 13784 heartbeater.cc:511] Master 127.31.32.62:38215 requested a full tablet report, sending...
W20260810 04:09:02.421778 13784 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 04:09:02.513862 13787 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:09:02.514139 13787 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:09:02.514173 13787 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:09:02.515444 13787 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:09:02.515512 13787 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:09:02.515558 13787 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.32.3
I20260810 04:09:02.517318 13787 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.32.20:33429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.31.32.20:53989/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.3.keytab
--principal=kudu/127.31.32.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.32.3:32797
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.32.3
--webserver_port=37617
--tserver_master_addrs=127.31.32.62:38215
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13787
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.32.3
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:09:02.517551 13787 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:09:02.517848 13787 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:09:02.524367 13795 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:09:02.524511 13787 server_base.cc:1034] running on GCE node
W20260810 04:09:02.524608 13793 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:09:02.524674 13792 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:09:02.524940 13787 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:09:02.525208 13787 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:09:02.526396 13787 hybrid_clock.cc:648] HybridClock initialized: now 1786334942526372 us; error 42 us; skew 500 ppm
Aug 10 04:09:02 dist-test-slave-8w9v krb5kdc[12299](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334942, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:09:02.529721 13787 init.cc:377] Logged in from keytab as kudu/127.31.32.3@KRBTEST.COM (short username kudu)
I20260810 04:09:02.530246 13787 webserver.cc:460] Webserver started at http://127.31.32.3:37617/ using document root <none> and password file <none>
I20260810 04:09:02.530506 13787 fs_manager.cc:359] Metadata directory not provided
I20260810 04:09:02.530591 13787 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 10 04:09:02 dist-test-slave-8w9v krb5kdc[12299](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334942, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.3@KRBTEST.COM for HTTP/127.31.32.20@KRBTEST.COM
I20260810 04:09:02.549928 13787 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.000s
I20260810 04:09:02.552351 13805 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:09:02.554646 13787 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.000s
I20260810 04:09:02.554754 13787 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/minicluster-data/ts-2/data,/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/minicluster-data/ts-2/wal
uuid: "fa3ec6a7a2584d78a3a51ebf25bee9c0"
format_stamp: "Formatted at 2026-08-10 04:09:01 on dist-test-slave-8w9v"
server_key: "71f1bce9141c11a3a7907be824f07ae2"
server_key_iv: "aad8a59951cfb6f6eadb9b2c61d37556"
server_key_version: "kuduclusterkey@0"
I20260810 04:09:02.555241 13787 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:09:02.573350 13787 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:09:02.575248 13787 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:09:02.575416 13787 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:09:02.575688 13787 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:09:02.576061 13787 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:09:02.576113 13787 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:09:02.576184 13787 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:09:02.576215 13787 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:09:02.584393 13787 rpc_server.cc:307] RPC server started. Bound to: 127.31.32.3:32797
I20260810 04:09:02.584882 13787 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/minicluster-data/ts-2/data/info.pb
I20260810 04:09:02.585424 13931 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.32.3:32797 every 8 connection(s)
I20260810 04:09:02.591827 31872 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskws89Io/build/release/bin/kudu as pid 13787
Aug 10 04:09:02 dist-test-slave-8w9v krb5kdc[12299](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334942, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.3@KRBTEST.COM for kudu/127.31.32.62@KRBTEST.COM
I20260810 04:09:02.599720 13639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:02.585755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.3:36151 (local address 127.31.32.62:38215)
0810 04:09:02.585814 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0810 04:09:02.585816 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:09:02.586140 (+ 324us) server_negotiation.cc:407] Connection header received
0810 04:09:02.586762 (+ 622us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:09:02.586764 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:09:02.586776 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:09:02.586857 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:09:02.587548 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:09:02.588370 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:09:02.589902 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:09:02.590207 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:09:02.590257 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:09:02.596461 (+ 6204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:09:02.596463 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:09:02.596464 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:09:02.596471 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:09:02.598344 (+ 1873us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:09:02.598749 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:09:02.598750 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:09:02.598751 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:09:02.598809 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:09:02.599067 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:09:02.599068 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:09:02.599068 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:09:02.599181 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:09:02.599253 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0810 04:09:02.599530 (+ 277us) server_negotiation.cc:299] Negotiation successful
0810 04:09:02.599629 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260810 04:09:02.599934 13934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:02.585754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:38215 (local address 127.31.32.3:36151)
0810 04:09:02.585982 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0810 04:09:02.586030 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0810 04:09:02.586647 (+ 617us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:09:02.586936 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:09:02.586941 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:09:02.587180 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:09:02.587441 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:09:02.587448 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:09:02.588490 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:09:02.588491 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:09:02.589787 (+ 1296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:09:02.589791 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:09:02.590377 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:09:02.590378 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:09:02.590514 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:09:02.591073 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:09:02.591084 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:09:02.596354 (+ 5270us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:09:02.598431 (+ 2077us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:09:02.598439 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:09:02.598441 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:09:02.598670 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:09:02.598882 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:09:02.598883 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:09:02.598883 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:09:02.599001 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:09:02.599244 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:09:02.599250 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:09:02.599400 (+ 150us) client_negotiation.cc:769] Sending connection context
0810 04:09:02.599507 (+ 107us) client_negotiation.cc:240] Negotiation successful
0810 04:09:02.599667 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":117,"threads_started":1}
I20260810 04:09:02.600385 13932 heartbeater.cc:347] Connected to a master server at 127.31.32.62:38215
I20260810 04:09:02.600520 13932 heartbeater.cc:464] Registering TS with master...
I20260810 04:09:02.600741 13932 heartbeater.cc:511] Master 127.31.32.62:38215 requested a full tablet report, sending...
W20260810 04:09:02.601398 13932 heartbeater.cc:644] Failed to heartbeat to 127.31.32.62:38215 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260810 04:09:02.601476 13932 heartbeater.cc:464] Registering TS with master...
I20260810 04:09:02.601548 13932 heartbeater.cc:511] Master 127.31.32.62:38215 requested a full tablet report, sending...
I20260810 04:09:02.601828 13932 heartbeater.cc:464] Registering TS with master...
I20260810 04:09:02.601912 13932 heartbeater.cc:511] Master 127.31.32.62:38215 requested a full tablet report, sending...
W20260810 04:09:02.602187 13932 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260810 04:09:02.606371 13935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:02.592265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47450 (local address 127.31.32.62:38215)
0810 04:09:02.593386 (+ 1121us) server_negotiation.cc:206] Beginning negotiation
0810 04:09:02.593388 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:09:02.593395 (+ 7us) server_negotiation.cc:407] Connection header received
0810 04:09:02.593415 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:09:02.593416 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:09:02.593428 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:09:02.593492 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:09:02.594615 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:09:02.595508 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:09:02.596632 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:09:02.596913 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:09:02.596982 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:09:02.600175 (+ 3193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:09:02.600176 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:09:02.600177 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:09:02.600184 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:09:02.605245 (+ 5061us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:09:02.605599 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:09:02.605600 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:09:02.605600 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:09:02.605657 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:09:02.605856 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:09:02.605857 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:09:02.605857 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:09:02.605966 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:09:02.606140 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0810 04:09:02.606176 (+ 36us) server_negotiation.cc:299] Negotiation successful
0810 04:09:02.606265 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1093,"thread_start_us":88,"threads_started":1}
Aug 10 04:09:03 dist-test-slave-8w9v krb5kdc[12299](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786334943, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:09:03.254264 13635 heartbeater.cc:464] Registering TS with master...
I20260810 04:09:03.254446 13635 heartbeater.cc:511] Master 127.31.32.62:38215 requested a full tablet report, sending...
I20260810 04:09:03.254915 13419 ts_manager.cc:194] Registered new tserver with Master: 3fd89e4040e5434788438b468df8e19c (127.31.32.1:36929)
I20260810 04:09:03.255942 13419 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.32.1@KRBTEST.COM'} at 127.31.32.1:55823
I20260810 04:09:03.421973 13784 heartbeater.cc:464] Registering TS with master...
I20260810 04:09:03.422138 13784 heartbeater.cc:511] Master 127.31.32.62:38215 requested a full tablet report, sending...
I20260810 04:09:03.422572 13419 ts_manager.cc:194] Registered new tserver with Master: 89e088f695d34b37997b7449193be12a (127.31.32.2:42877)
I20260810 04:09:03.423117 13419 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.32.2@KRBTEST.COM'} at 127.31.32.2:56993
Aug 10 04:09:03 dist-test-slave-8w9v krb5kdc[12299](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334943, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.62@KRBTEST.COM for HTTP/127.31.32.20@KRBTEST.COM
I20260810 04:09:03.602417 13932 heartbeater.cc:464] Registering TS with master...
I20260810 04:09:03.602586 13932 heartbeater.cc:511] Master 127.31.32.62:38215 requested a full tablet report, sending...
I20260810 04:09:03.602926 13419 ts_manager.cc:194] Registered new tserver with Master: fa3ec6a7a2584d78a3a51ebf25bee9c0 (127.31.32.3:32797)
I20260810 04:09:03.603434 13419 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.32.3@KRBTEST.COM'} at 127.31.32.3:36151
I20260810 04:09:03.610037 31872 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 04:09:03.620337 13955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:03.611183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41240 (local address 127.31.32.62:38215)
0810 04:09:03.611333 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0810 04:09:03.611335 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:09:03.612382 (+ 1047us) server_negotiation.cc:407] Connection header received
0810 04:09:03.612438 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:09:03.612439 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:09:03.612456 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:09:03.612539 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:09:03.613232 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:09:03.614005 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:09:03.615056 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:09:03.615359 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:09:03.615440 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:09:03.616959 (+ 1519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:09:03.616961 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:09:03.616961 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:09:03.616969 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:09:03.618827 (+ 1858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:09:03.619200 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:09:03.619201 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:09:03.619202 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:09:03.619281 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:09:03.619532 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:09:03.619533 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:09:03.619533 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:09:03.619665 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:09:03.619756 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0810 04:09:03.620153 (+ 397us) server_negotiation.cc:299] Negotiation successful
0810 04:09:03.620235 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":81,"threads_started":1}
I20260810 04:09:03.622836 13419 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41240:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 04:09:03.625023 13419 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 04:09:03.634142 13962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:03.628281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.3:32797 (local address 127.0.0.1:55344)
0810 04:09:03.628716 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0810 04:09:03.628761 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 04:09:03.628837 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:09:03.629423 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:09:03.629426 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:09:03.629429 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:09:03.629500 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:09:03.629502 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:09:03.630768 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:09:03.630769 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:09:03.632409 (+ 1640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:09:03.632412 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:09:03.633812 (+ 1400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:09:03.633813 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:09:03.633901 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:09:03.633915 (+ 14us) client_negotiation.cc:769] Sending connection context
0810 04:09:03.633990 (+ 75us) client_negotiation.cc:240] Negotiation successful
0810 04:09:03.634046 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":96,"threads_started":1}
I20260810 04:09:03.634333 13966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:03.629066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48794 (local address 127.31.32.1:36929)
0810 04:09:03.629538 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0810 04:09:03.629542 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 04:09:03.629664 (+ 122us) server_negotiation.cc:407] Connection header received
0810 04:09:03.630269 (+ 605us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:09:03.630272 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:09:03.630287 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:09:03.630360 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:09:03.630565 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:09:03.631575 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:09:03.633041 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:09:03.633805 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:09:03.633888 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:09:03.633960 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0810 04:09:03.634185 (+ 225us) server_negotiation.cc:299] Negotiation successful
0810 04:09:03.634226 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":437,"thread_start_us":69,"threads_started":1}
I20260810 04:09:03.634625 13960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:03.628281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.1:36929 (local address 127.0.0.1:48794)
0810 04:09:03.629600 (+ 1319us) negotiation.cc:107] Waiting for socket to connect
0810 04:09:03.629611 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 04:09:03.630147 (+ 536us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:09:03.630449 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:09:03.630450 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:09:03.630452 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:09:03.630491 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:09:03.630492 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:09:03.631807 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:09:03.631808 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:09:03.632937 (+ 1129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:09:03.632940 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:09:03.633995 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:09:03.633996 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:09:03.634049 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:09:03.634052 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 04:09:03.634104 (+ 52us) client_negotiation.cc:240] Negotiation successful
0810 04:09:03.634137 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1301,"mutex_wait_us":3,"thread_start_us":92,"threads_started":1}
I20260810 04:09:03.634917 13965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:03.628843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55344 (local address 127.31.32.3:32797)
0810 04:09:03.629250 (+ 407us) server_negotiation.cc:206] Beginning negotiation
0810 04:09:03.629255 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 04:09:03.629268 (+ 13us) server_negotiation.cc:407] Connection header received
0810 04:09:03.629307 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:09:03.629309 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:09:03.629327 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:09:03.629419 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:09:03.629598 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:09:03.630670 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:09:03.632515 (+ 1845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:09:03.633626 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:09:03.633698 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:09:03.633760 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0810 04:09:03.634722 (+ 962us) server_negotiation.cc:299] Negotiation successful
0810 04:09:03.634835 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":68,"threads_started":1}
I20260810 04:09:03.635705 13961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:03.628541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.2:42877 (local address 127.0.0.1:37176)
0810 04:09:03.629810 (+ 1269us) negotiation.cc:107] Waiting for socket to connect
0810 04:09:03.629820 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 04:09:03.629876 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:09:03.631679 (+ 1803us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:09:03.631680 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:09:03.631682 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:09:03.631718 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:09:03.631720 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:09:03.633338 (+ 1618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:09:03.633339 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:09:03.634374 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:09:03.634377 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:09:03.635466 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:09:03.635467 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:09:03.635546 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:09:03.635550 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 04:09:03.635598 (+ 48us) client_negotiation.cc:240] Negotiation successful
0810 04:09:03.635641 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1253,"thread_start_us":105,"threads_started":1}
I20260810 04:09:03.635749 13837 tablet_service.cc:1467] Processing CreateTablet for tablet ce2455ad99f849008fae7969ab1b8a06 (DEFAULT_TABLE table=test-table [id=6510f22ab1954377afeb735d4b05afba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:09:03.636039 13964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:03.628642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37176 (local address 127.31.32.2:42877)
0810 04:09:03.629753 (+ 1111us) server_negotiation.cc:206] Beginning negotiation
0810 04:09:03.629757 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 04:09:03.629939 (+ 182us) server_negotiation.cc:407] Connection header received
0810 04:09:03.629977 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:09:03.629979 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:09:03.629995 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:09:03.630068 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:09:03.632197 (+ 2129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:09:03.633198 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:09:03.634477 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:09:03.635294 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:09:03.635380 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:09:03.635430 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0810 04:09:03.635912 (+ 482us) server_negotiation.cc:299] Negotiation successful
0810 04:09:03.635952 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1064,"thread_start_us":50,"threads_started":1}
I20260810 04:09:03.636016 13564 tablet_service.cc:1467] Processing CreateTablet for tablet ce2455ad99f849008fae7969ab1b8a06 (DEFAULT_TABLE table=test-table [id=6510f22ab1954377afeb735d4b05afba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:09:03.636211 13837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce2455ad99f849008fae7969ab1b8a06. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:09:03.636356 13564 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce2455ad99f849008fae7969ab1b8a06. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:09:03.639895 13967 tablet_bootstrap.cc:492] T ce2455ad99f849008fae7969ab1b8a06 P 3fd89e4040e5434788438b468df8e19c: Bootstrap starting.
I20260810 04:09:03.640607 13967 tablet_bootstrap.cc:654] T ce2455ad99f849008fae7969ab1b8a06 P 3fd89e4040e5434788438b468df8e19c: Neither blocks nor log segments found. Creating new log.
I20260810 04:09:03.640909 13700 tablet_service.cc:1467] Processing CreateTablet for tablet ce2455ad99f849008fae7969ab1b8a06 (DEFAULT_TABLE table=test-table [id=6510f22ab1954377afeb735d4b05afba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:09:03.640974 13967 log.cc:824] T ce2455ad99f849008fae7969ab1b8a06 P 3fd89e4040e5434788438b468df8e19c: Log is configured to *not* fsync() on all Append() calls
I20260810 04:09:03.641243 13700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce2455ad99f849008fae7969ab1b8a06. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:09:03.641350 13969 tablet_bootstrap.cc:492] T ce2455ad99f849008fae7969ab1b8a06 P fa3ec6a7a2584d78a3a51ebf25bee9c0: Bootstrap starting.
I20260810 04:09:03.641791 13967 tablet_bootstrap.cc:492] T ce2455ad99f849008fae7969ab1b8a06 P 3fd89e4040e5434788438b468df8e19c: No bootstrap required, opened a new log
I20260810 04:09:03.641851 13967 ts_tablet_manager.cc:1397] T ce2455ad99f849008fae7969ab1b8a06 P 3fd89e4040e5434788438b468df8e19c: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260810 04:09:03.642009 13969 tablet_bootstrap.cc:654] T ce2455ad99f849008fae7969ab1b8a06 P fa3ec6a7a2584d78a3a51ebf25bee9c0: Neither blocks nor log segments found. Creating new log.
I20260810 04:09:03.642313 13969 log.cc:824] T ce2455ad99f849008fae7969ab1b8a06 P fa3ec6a7a2584d78a3a51ebf25bee9c0: Log is configured to *not* fsync() on all Append() calls
I20260810 04:09:03.644277 13967 raft_consensus.cc:348] T ce2455ad99f849008fae7969ab1b8a06 P 3fd89e4040e5434788438b468df8e19c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fd89e4040e5434788438b468df8e19c" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 36929 } } peers { permanent_uuid: "89e088f695d34b37997b7449193be12a" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 42877 } } peers { permanent_uuid: "fa3ec6a7a2584d78a3a51ebf25bee9c0" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 32797 } }
I20260810 04:09:03.644465 13967 raft_consensus.cc:374] T ce2455ad99f849008fae7969ab1b8a06 P 3fd89e4040e5434788438b468df8e19c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:09:03.644546 13967 raft_consensus.cc:729] T ce2455ad99f849008fae7969ab1b8a06 P 3fd89e4040e5434788438b468df8e19c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fd89e4040e5434788438b468df8e19c, State: Initialized, Role: FOLLOWER
I20260810 04:09:03.644698 13967 consensus_queue.cc:260] T ce2455ad99f849008fae7969ab1b8a06 P 3fd89e4040e5434788438b468df8e19c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fd89e4040e5434788438b468df8e19c" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 36929 } } peers { permanent_uuid: "89e088f695d34b37997b7449193be12a" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 42877 } } peers { permanent_uuid: "fa3ec6a7a2584d78a3a51ebf25bee9c0" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 32797 } }
I20260810 04:09:03.644866 13972 tablet_bootstrap.cc:492] T ce2455ad99f849008fae7969ab1b8a06 P 89e088f695d34b37997b7449193be12a: Bootstrap starting.
I20260810 04:09:03.645285 13635 heartbeater.cc:503] Master 127.31.32.62:38215 was elected leader, sending a full tablet report...
I20260810 04:09:03.645493 13972 tablet_bootstrap.cc:654] T ce2455ad99f849008fae7969ab1b8a06 P 89e088f695d34b37997b7449193be12a: Neither blocks nor log segments found. Creating new log.
I20260810 04:09:03.645269 13967 ts_tablet_manager.cc:1428] T ce2455ad99f849008fae7969ab1b8a06 P 3fd89e4040e5434788438b468df8e19c: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260810 04:09:03.645953 13972 log.cc:824] T ce2455ad99f849008fae7969ab1b8a06 P 89e088f695d34b37997b7449193be12a: Log is configured to *not* fsync() on all Append() calls
I20260810 04:09:03.646234 13969 tablet_bootstrap.cc:492] T ce2455ad99f849008fae7969ab1b8a06 P fa3ec6a7a2584d78a3a51ebf25bee9c0: No bootstrap required, opened a new log
I20260810 04:09:03.646315 13969 ts_tablet_manager.cc:1397] T ce2455ad99f849008fae7969ab1b8a06 P fa3ec6a7a2584d78a3a51ebf25bee9c0: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260810 04:09:03.648247 13969 raft_consensus.cc:348] T ce2455ad99f849008fae7969ab1b8a06 P fa3ec6a7a2584d78a3a51ebf25bee9c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fd89e4040e5434788438b468df8e19c" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 36929 } } peers { permanent_uuid: "89e088f695d34b37997b7449193be12a" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 42877 } } peers { permanent_uuid: "fa3ec6a7a2584d78a3a51ebf25bee9c0" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 32797 } }
I20260810 04:09:03.648443 13969 raft_consensus.cc:374] T ce2455ad99f849008fae7969ab1b8a06 P fa3ec6a7a2584d78a3a51ebf25bee9c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:09:03.648480 13969 raft_consensus.cc:729] T ce2455ad99f849008fae7969ab1b8a06 P fa3ec6a7a2584d78a3a51ebf25bee9c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa3ec6a7a2584d78a3a51ebf25bee9c0, State: Initialized, Role: FOLLOWER
I20260810 04:09:03.648595 13969 consensus_queue.cc:260] T ce2455ad99f849008fae7969ab1b8a06 P fa3ec6a7a2584d78a3a51ebf25bee9c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fd89e4040e5434788438b468df8e19c" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 36929 } } peers { permanent_uuid: "89e088f695d34b37997b7449193be12a" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 42877 } } peers { permanent_uuid: "fa3ec6a7a2584d78a3a51ebf25bee9c0" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 32797 } }
I20260810 04:09:03.648761 13972 tablet_bootstrap.cc:492] T ce2455ad99f849008fae7969ab1b8a06 P 89e088f695d34b37997b7449193be12a: No bootstrap required, opened a new log
I20260810 04:09:03.648833 13972 ts_tablet_manager.cc:1397] T ce2455ad99f849008fae7969ab1b8a06 P 89e088f695d34b37997b7449193be12a: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260810 04:09:03.648849 13969 ts_tablet_manager.cc:1428] T ce2455ad99f849008fae7969ab1b8a06 P fa3ec6a7a2584d78a3a51ebf25bee9c0: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260810 04:09:03.648940 13932 heartbeater.cc:503] Master 127.31.32.62:38215 was elected leader, sending a full tablet report...
I20260810 04:09:03.651268 13972 raft_consensus.cc:348] T ce2455ad99f849008fae7969ab1b8a06 P 89e088f695d34b37997b7449193be12a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fd89e4040e5434788438b468df8e19c" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 36929 } } peers { permanent_uuid: "89e088f695d34b37997b7449193be12a" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 42877 } } peers { permanent_uuid: "fa3ec6a7a2584d78a3a51ebf25bee9c0" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 32797 } }
I20260810 04:09:03.651420 13972 raft_consensus.cc:374] T ce2455ad99f849008fae7969ab1b8a06 P 89e088f695d34b37997b7449193be12a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:09:03.651450 13972 raft_consensus.cc:729] T ce2455ad99f849008fae7969ab1b8a06 P 89e088f695d34b37997b7449193be12a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89e088f695d34b37997b7449193be12a, State: Initialized, Role: FOLLOWER
I20260810 04:09:03.651549 13972 consensus_queue.cc:260] T ce2455ad99f849008fae7969ab1b8a06 P 89e088f695d34b37997b7449193be12a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fd89e4040e5434788438b468df8e19c" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 36929 } } peers { permanent_uuid: "89e088f695d34b37997b7449193be12a" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 42877 } } peers { permanent_uuid: "fa3ec6a7a2584d78a3a51ebf25bee9c0" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 32797 } }
I20260810 04:09:03.651814 13972 ts_tablet_manager.cc:1428] T ce2455ad99f849008fae7969ab1b8a06 P 89e088f695d34b37997b7449193be12a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 04:09:03.651852 13784 heartbeater.cc:503] Master 127.31.32.62:38215 was elected leader, sending a full tablet report...
W20260810 04:09:03.730270 13636 tablet.cc:2369] T ce2455ad99f849008fae7969ab1b8a06 P 3fd89e4040e5434788438b468df8e19c: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 10 04:09:03 dist-test-slave-8w9v krb5kdc[12299](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334943, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.62@KRBTEST.COM for HTTP/127.31.32.20@KRBTEST.COM
W20260810 04:09:03.838115 13933 tablet.cc:2369] T ce2455ad99f849008fae7969ab1b8a06 P fa3ec6a7a2584d78a3a51ebf25bee9c0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 04:09:03.854375 13976 raft_consensus.cc:482] T ce2455ad99f849008fae7969ab1b8a06 P 89e088f695d34b37997b7449193be12a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 04:09:03.854552 13976 raft_consensus.cc:504] T ce2455ad99f849008fae7969ab1b8a06 P 89e088f695d34b37997b7449193be12a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fd89e4040e5434788438b468df8e19c" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 36929 } } peers { permanent_uuid: "89e088f695d34b37997b7449193be12a" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 42877 } } peers { permanent_uuid: "fa3ec6a7a2584d78a3a51ebf25bee9c0" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 32797 } }
I20260810 04:09:03.854972 13976 leader_election.cc:288] T ce2455ad99f849008fae7969ab1b8a06 P 89e088f695d34b37997b7449193be12a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fd89e4040e5434788438b468df8e19c (127.31.32.1:36929), fa3ec6a7a2584d78a3a51ebf25bee9c0 (127.31.32.3:32797)
I20260810 04:09:03.859043 13966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:03.855480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.2:52823 (local address 127.31.32.1:36929)
0810 04:09:03.855525 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0810 04:09:03.855527 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:09:03.855536 (+ 9us) server_negotiation.cc:407] Connection header received
0810 04:09:03.855567 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:09:03.855567 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:09:03.855579 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:09:03.855623 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:09:03.855817 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:09:03.856536 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:09:03.858001 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:09:03.858571 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:09:03.858620 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:09:03.858632 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0810 04:09:03.858926 (+ 294us) server_negotiation.cc:299] Negotiation successful
0810 04:09:03.858972 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260810 04:09:03.859297 13979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:03.855175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.1:36929 (local address 127.31.32.2:52823)
0810 04:09:03.855328 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0810 04:09:03.855340 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 04:09:03.855393 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:09:03.855641 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:09:03.855642 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:09:03.855644 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:09:03.855733 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:09:03.855735 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:09:03.856638 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:09:03.856639 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:09:03.857895 (+ 1256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:09:03.857898 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:09:03.858766 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:09:03.858767 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:09:03.858840 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:09:03.858845 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 04:09:03.859162 (+ 317us) client_negotiation.cc:240] Negotiation successful
0810 04:09:03.859204 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":97,"threads_started":1}
I20260810 04:09:03.859632 13577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce2455ad99f849008fae7969ab1b8a06" candidate_uuid: "89e088f695d34b37997b7449193be12a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fd89e4040e5434788438b468df8e19c" is_pre_election: true
I20260810 04:09:03.859812 13577 raft_consensus.cc:2454] T ce2455ad99f849008fae7969ab1b8a06 P 3fd89e4040e5434788438b468df8e19c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89e088f695d34b37997b7449193be12a in term 0.
I20260810 04:09:03.860051 13661 leader_election.cc:302] T ce2455ad99f849008fae7969ab1b8a06 P 89e088f695d34b37997b7449193be12a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fd89e4040e5434788438b468df8e19c, 89e088f695d34b37997b7449193be12a; no voters:
I20260810 04:09:03.860203 13976 raft_consensus.cc:2781] T ce2455ad99f849008fae7969ab1b8a06 P 89e088f695d34b37997b7449193be12a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 04:09:03.860248 13976 raft_consensus.cc:482] T ce2455ad99f849008fae7969ab1b8a06 P 89e088f695d34b37997b7449193be12a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 04:09:03.860271 13976 raft_consensus.cc:3037] T ce2455ad99f849008fae7969ab1b8a06 P 89e088f695d34b37997b7449193be12a [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:09:03.860329 13965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:03.855425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.2:57465 (local address 127.31.32.3:32797)
0810 04:09:03.855530 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0810 04:09:03.855532 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:09:03.855836 (+ 304us) server_negotiation.cc:407] Connection header received
0810 04:09:03.855934 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:09:03.855935 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:09:03.855947 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:09:03.855975 (+ 28us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:09:03.856223 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:09:03.856943 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:09:03.858486 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:09:03.859090 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:09:03.859136 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:09:03.859145 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0810 04:09:03.860207 (+ 1062us) server_negotiation.cc:299] Negotiation successful
0810 04:09:03.860252 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260810 04:09:03.860548 13980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:03.855190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.3:32797 (local address 127.31.32.2:57465)
0810 04:09:03.855771 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0810 04:09:03.855782 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 04:09:03.855864 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:09:03.856046 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:09:03.856047 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:09:03.856048 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:09:03.856142 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:09:03.856144 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:09:03.857073 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:09:03.857074 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:09:03.858364 (+ 1290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:09:03.858367 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:09:03.860030 (+ 1663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:09:03.860031 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:09:03.860102 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:09:03.860105 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 04:09:03.860406 (+ 301us) client_negotiation.cc:240] Negotiation successful
0810 04:09:03.860447 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":561,"thread_start_us":108,"threads_started":1}
I20260810 04:09:03.860925 13876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce2455ad99f849008fae7969ab1b8a06" candidate_uuid: "89e088f695d34b37997b7449193be12a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa3ec6a7a2584d78a3a51ebf25bee9c0" is_pre_election: true
I20260810 04:09:03.861088 13876 raft_consensus.cc:2454] T ce2455ad99f849008fae7969ab1b8a06 P fa3ec6a7a2584d78a3a51ebf25bee9c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89e088f695d34b37997b7449193be12a in term 0.
I20260810 04:09:03.861231 13976 raft_consensus.cc:504] T ce2455ad99f849008fae7969ab1b8a06 P 89e088f695d34b37997b7449193be12a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fd89e4040e5434788438b468df8e19c" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 36929 } } peers { permanent_uuid: "89e088f695d34b37997b7449193be12a" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 42877 } } peers { permanent_uuid: "fa3ec6a7a2584d78a3a51ebf25bee9c0" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 32797 } }
I20260810 04:09:03.861383 13976 leader_election.cc:288] T ce2455ad99f849008fae7969ab1b8a06 P 89e088f695d34b37997b7449193be12a [CANDIDATE]: Term 1 election: Requested vote from peers 3fd89e4040e5434788438b468df8e19c (127.31.32.1:36929), fa3ec6a7a2584d78a3a51ebf25bee9c0 (127.31.32.3:32797)
I20260810 04:09:03.861524 13876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce2455ad99f849008fae7969ab1b8a06" candidate_uuid: "89e088f695d34b37997b7449193be12a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa3ec6a7a2584d78a3a51ebf25bee9c0"
I20260810 04:09:03.861608 13876 raft_consensus.cc:3037] T ce2455ad99f849008fae7969ab1b8a06 P fa3ec6a7a2584d78a3a51ebf25bee9c0 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:09:03.861649 13577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce2455ad99f849008fae7969ab1b8a06" candidate_uuid: "89e088f695d34b37997b7449193be12a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fd89e4040e5434788438b468df8e19c"
I20260810 04:09:03.861724 13577 raft_consensus.cc:3037] T ce2455ad99f849008fae7969ab1b8a06 P 3fd89e4040e5434788438b468df8e19c [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:09:03.862509 13577 raft_consensus.cc:2454] T ce2455ad99f849008fae7969ab1b8a06 P 3fd89e4040e5434788438b468df8e19c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89e088f695d34b37997b7449193be12a in term 1.
I20260810 04:09:03.862572 13876 raft_consensus.cc:2454] T ce2455ad99f849008fae7969ab1b8a06 P fa3ec6a7a2584d78a3a51ebf25bee9c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89e088f695d34b37997b7449193be12a in term 1.
I20260810 04:09:03.862679 13661 leader_election.cc:302] T ce2455ad99f849008fae7969ab1b8a06 P 89e088f695d34b37997b7449193be12a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fd89e4040e5434788438b468df8e19c, 89e088f695d34b37997b7449193be12a; no voters:
I20260810 04:09:03.862768 13976 raft_consensus.cc:2781] T ce2455ad99f849008fae7969ab1b8a06 P 89e088f695d34b37997b7449193be12a [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:09:03.863042 13976 raft_consensus.cc:686] T ce2455ad99f849008fae7969ab1b8a06 P 89e088f695d34b37997b7449193be12a [term 1 LEADER]: Becoming Leader. State: Replica: 89e088f695d34b37997b7449193be12a, State: Running, Role: LEADER
I20260810 04:09:03.863138 13976 consensus_queue.cc:237] T ce2455ad99f849008fae7969ab1b8a06 P 89e088f695d34b37997b7449193be12a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fd89e4040e5434788438b468df8e19c" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 36929 } } peers { permanent_uuid: "89e088f695d34b37997b7449193be12a" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 42877 } } peers { permanent_uuid: "fa3ec6a7a2584d78a3a51ebf25bee9c0" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 32797 } }
I20260810 04:09:03.864054 13419 catalog_manager.cc:5495] T ce2455ad99f849008fae7969ab1b8a06 P 89e088f695d34b37997b7449193be12a reported cstate change: term changed from 0 to 1, leader changed from <none> to 89e088f695d34b37997b7449193be12a (127.31.32.2). New cstate: current_term: 1 leader_uuid: "89e088f695d34b37997b7449193be12a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fd89e4040e5434788438b468df8e19c" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 36929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89e088f695d34b37997b7449193be12a" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 42877 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fa3ec6a7a2584d78a3a51ebf25bee9c0" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 32797 } health_report { overall_health: UNKNOWN } } }
W20260810 04:09:03.900286 13785 tablet.cc:2369] T ce2455ad99f849008fae7969ab1b8a06 P 89e088f695d34b37997b7449193be12a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 04:09:03.921096 13964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:03.917985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37190 (local address 127.31.32.2:42877)
0810 04:09:03.918048 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0810 04:09:03.918050 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:09:03.918060 (+ 10us) server_negotiation.cc:407] Connection header received
0810 04:09:03.918101 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:09:03.918102 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:09:03.918124 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:09:03.918189 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 04:09:03.918340 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:09:03.918803 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:09:03.920006 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:09:03.920288 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:09:03.920363 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:09:03.920541 (+ 178us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 04:09:03.920670 (+ 129us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 04:09:03.920753 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0810 04:09:03.920995 (+ 242us) server_negotiation.cc:299] Negotiation successful
0810 04:09:03.921018 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260810 04:09:03.923141 13876 raft_consensus.cc:1261] T ce2455ad99f849008fae7969ab1b8a06 P fa3ec6a7a2584d78a3a51ebf25bee9c0 [term 1 FOLLOWER]: Refusing update from remote peer 89e088f695d34b37997b7449193be12a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:09:03.923491 13976 consensus_queue.cc:1035] T ce2455ad99f849008fae7969ab1b8a06 P 89e088f695d34b37997b7449193be12a [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa3ec6a7a2584d78a3a51ebf25bee9c0" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 32797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:09:03.923565 13577 raft_consensus.cc:1261] T ce2455ad99f849008fae7969ab1b8a06 P 3fd89e4040e5434788438b468df8e19c [term 1 FOLLOWER]: Refusing update from remote peer 89e088f695d34b37997b7449193be12a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:09:03.923836 13976 consensus_queue.cc:1035] T ce2455ad99f849008fae7969ab1b8a06 P 89e088f695d34b37997b7449193be12a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fd89e4040e5434788438b468df8e19c" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 36929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:09:03.926626 13984 mvcc.cc:205] Tried to move back new op lower bound from 7316827930306117632 to 7316827930063835136. Current Snapshot: MvccSnapshot[applied={T|T < 7316827930306117632}]
I20260810 04:09:03.927942 13985 mvcc.cc:205] Tried to move back new op lower bound from 7316827930306117632 to 7316827930063835136. Current Snapshot: MvccSnapshot[applied={T|T < 7316827930306117632}]
I20260810 04:09:03.928989 13417 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41240:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 04:09:03.929076 13417 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41240:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 04:09:03.930454 13417 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b615bc77cb4f4ad59550be465d809fb3: Sending DeleteTablet for 3 replicas of tablet ce2455ad99f849008fae7969ab1b8a06
I20260810 04:09:03.930802 13564 tablet_service.cc:1514] Processing DeleteTablet for tablet ce2455ad99f849008fae7969ab1b8a06 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:09:03 UTC) from {username='kudu', principal='kudu/127.31.32.62@KRBTEST.COM'} at 127.0.0.1:48794
I20260810 04:09:03.931169 13837 tablet_service.cc:1514] Processing DeleteTablet for tablet ce2455ad99f849008fae7969ab1b8a06 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:09:03 UTC) from {username='kudu', principal='kudu/127.31.32.62@KRBTEST.COM'} at 127.0.0.1:55344
I20260810 04:09:03.931452 13700 tablet_service.cc:1514] Processing DeleteTablet for tablet ce2455ad99f849008fae7969ab1b8a06 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:09:03 UTC) from {username='kudu', principal='kudu/127.31.32.62@KRBTEST.COM'} at 127.0.0.1:37176
I20260810 04:09:03.932436 31872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskws89Io/build/release/bin/kudu with pid 13460
I20260810 04:09:03.937948 13986 mvcc.cc:205] Tried to move back new op lower bound from 7316827930306117632 to 7316827930063835136. Current Snapshot: MvccSnapshot[applied={T|T < 7316827930306117632}]
I20260810 04:09:03.938202 13993 tablet_replica.cc:330] T ce2455ad99f849008fae7969ab1b8a06 P fa3ec6a7a2584d78a3a51ebf25bee9c0: stopping tablet replica
I20260810 04:09:03.938325 13994 tablet_replica.cc:330] T ce2455ad99f849008fae7969ab1b8a06 P 89e088f695d34b37997b7449193be12a: stopping tablet replica
I20260810 04:09:03.938447 13993 raft_consensus.cc:2229] T ce2455ad99f849008fae7969ab1b8a06 P fa3ec6a7a2584d78a3a51ebf25bee9c0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 04:09:03.938647 13994 raft_consensus.cc:2229] T ce2455ad99f849008fae7969ab1b8a06 P 89e088f695d34b37997b7449193be12a [term 1 LEADER]: Raft consensus shutting down.
I20260810 04:09:03.938767 13994 raft_consensus.cc:2258] T ce2455ad99f849008fae7969ab1b8a06 P 89e088f695d34b37997b7449193be12a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:09:03.938881 13993 raft_consensus.cc:2258] T ce2455ad99f849008fae7969ab1b8a06 P fa3ec6a7a2584d78a3a51ebf25bee9c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:09:03.941336 13994 ts_tablet_manager.cc:1905] T ce2455ad99f849008fae7969ab1b8a06 P 89e088f695d34b37997b7449193be12a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:09:03.943398 13994 ts_tablet_manager.cc:1918] T ce2455ad99f849008fae7969ab1b8a06 P 89e088f695d34b37997b7449193be12a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 04:09:03.943521 13994 log.cc:1196] T ce2455ad99f849008fae7969ab1b8a06 P 89e088f695d34b37997b7449193be12a: Deleting WAL directory at /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786334794676589-31872-0/minicluster-data/ts-1/wal/wals/ce2455ad99f849008fae7969ab1b8a06
I20260810 04:09:03.943894 13994 ts_tablet_manager.cc:1939] T ce2455ad99f849008fae7969ab1b8a06 P 89e088f695d34b37997b7449193be12a: Deleting consensus metadata
I20260810 04:09:03.944413 13403 catalog_manager.cc:4841] TS 89e088f695d34b37997b7449193be12a (127.31.32.2:42877): tablet ce2455ad99f849008fae7969ab1b8a06 (table test-table [id=6510f22ab1954377afeb735d4b05afba]) successfully deleted
W20260810 04:09:03.945861 13403 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.32.1:36929 (error 108)
I20260810 04:09:03.946373 13960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:03.946238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.1:36929 (local address 127.0.0.1:48806)
0810 04:09:03.946285 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0810 04:09:03.946341 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.1:36929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260810 04:09:03.946581 13403 catalog_manager.cc:4584] TS 3fd89e4040e5434788438b468df8e19c (127.31.32.1:36929): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ce2455ad99f849008fae7969ab1b8a06: Network error: Client connection negotiation failed: client connection to 127.31.32.1:36929: connect: Connection refused (error 111)
I20260810 04:09:03.947031 31872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskws89Io/build/release/bin/kudu with pid 13637
I20260810 04:09:03.957706 31872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskws89Io/build/release/bin/kudu with pid 13787
W20260810 04:09:03.979858 13960 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:09:03.979544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.3:32797 (local address 127.0.0.1:55346)
0810 04:09:03.979597 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0810 04:09:03.979608 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 04:09:03.979652 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:09:03.979790 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.3:32797: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":32}
I20260810 04:09:03.980666 31872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskws89Io/build/release/bin/kudu with pid 13382
I20260810 04:09:03.999174 31872 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260810 04:09:04.343380 31872 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260810 04:09:04.343492 31872 mini_ranger.cc:66] Stopping Ranger...
I20260810 04:09:04.687865 31872 mini_ranger.cc:68] Stopped Ranger
2026-08-10 04:09:04.688 UTC [12332] LOG: received smart shutdown request
2026-08-10 04:09:04.692 UTC [12332] LOG: background worker "logical replication launcher" (PID 12342) exited with exit code 1
2026-08-10 04:09:04.693 UTC [12336] LOG: shutting down
2026-08-10 04:09:04.850 UTC [12332] LOG: database system is shut down
2026-08-10T04:09:04Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (46874 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786334794676589-31872-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 04:09:05 dist-test-slave-8w9v krb5kdc[14006](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 04:09:05 dist-test-slave-8w9v krb5kdc[14006](info): set up 2 sockets
Aug 10 04:09:05 dist-test-slave-8w9v krb5kdc[14006](info): commencing operation
krb5kdc: starting...
W20260810 04:09:06.057942 31872 mini_kdc.cc:121] Time spent starting KDC: real 1.069s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786334794676589-31872-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786334794676589-31872-0/krb5kdc/test-user.keytab.
Aug 10 04:09:06 dist-test-slave-8w9v krb5kdc[14006](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334946, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T04:09:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T04:09:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.32.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.32.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.32.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.62.keytab.
Entry for principal kudu/127.31.32.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.62.keytab.
I20260810 04:09:06.178944 31872 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskws89Io/build/release/bin/kudu
/tmp/dist-test-taskws89Io/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786334794676589-31872-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786334794676589-31872-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786334794676589-31872-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786334794676589-31872-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.32.62:45713
--webserver_interface=127.31.32.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.62.keytab
--principal=kudu/127.31.32.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.32.20:39979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.32.62:45713,127.31.32.61:38055,127.31.32.60:37637
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786334794676589-31872-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786334794676589-31872-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786334794676589-31872-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:09:06.261067 14020 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:09:06.261266 14020 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:09:06.261296 14020 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:09:06.262428 14020 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:09:06.262470 14020 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:09:06.262483 14020 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:09:06.262495 14020 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:09:06.262569 14020 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:09:06.264051 14020 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.32.20:39979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786334794676589-31872-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786334794676589-31872-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.32.62:45713,127.31.32.61:38055,127.31.32.60:37637
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.62.keytab
--principal=kudu/127.31.32.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.32.62:45713
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786334794676589-31872-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.32.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14020
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786334794676589-31872-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:09:06.264370 14020 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:09:06.264611 14020 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:09:06.269361 14027 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:09:06.269398 14026 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:09:06.269382 14020 server_base.cc:1034] running on GCE node
W20260810 04:09:06.269366 14029 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:09:06.269876 14020 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:09:06.270128 14020 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:09:06.271311 14020 hybrid_clock.cc:648] HybridClock initialized: now 1786334946271263 us; error 48 us; skew 500 ppm
Aug 10 04:09:06 dist-test-slave-8w9v krb5kdc[14006](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334946, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:09:06.277484 14020 init.cc:377] Logged in from keytab as kudu/127.31.32.62@KRBTEST.COM (short username kudu)
I20260810 04:09:06.277932 14020 webserver.cc:460] Webserver started at http://127.31.32.62:41947/ using document root <none> and password file <none>
I20260810 04:09:06.278137 14020 fs_manager.cc:359] Metadata directory not provided
I20260810 04:09:06.278178 14020 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:09:06.278264 14020 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:09:06.279287 14020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786334794676589-31872-0/minicluster-data/master-0/data/instance:
uuid: "44c973ce2d7748e0b843e447666bc8ca"
format_stamp: "Formatted at 2026-08-10 04:09:06 on dist-test-slave-8w9v"
server_key: "fcd8a5835f07132da9eff681ea295c4d"
server_key_iv: "ff291529c622dde2198bb983e54f8bd4"
server_key_version: "encryptionkey@0"
I20260810 04:09:06.279613 14020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786334794676589-31872-0/minicluster-data/master-0/wal/instance:
uuid: "44c973ce2d7748e0b843e447666bc8ca"
format_stamp: "Formatted at 2026-08-10 04:09:06 on dist-test-slave-8w9v"
server_key: "fcd8a5835f07132da9eff681ea295c4d"
server_key_iv: "ff291529c622dde2198bb983e54f8bd4"
server_key_version: "encryptionkey@0"
I20260810 04:09:06.280937 14020 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 04:09:06.281673 14037 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:09:06.281859 14020 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 04:09:06.281965 14020 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786334794676589-31872-0/minicluster-data/master-0/data,/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786334794676589-31872-0/minicluster-data/master-0/wal
uuid: "44c973ce2d7748e0b843e447666bc8ca"
format_stamp: "Formatted at 2026-08-10 04:09:06 on dist-test-slave-8w9v"
server_key: "fcd8a5835f07132da9eff681ea295c4d"
server_key_iv: "ff291529c622dde2198bb983e54f8bd4"
server_key_version: "encryptionkey@0"
I20260810 04:09:06.282035 14020 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786334794676589-31872-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786334794676589-31872-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786334794676589-31872-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:09:06.301759 14020 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:09:06.304117 14020 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:09:06.304301 14020 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:09:06.308768 14020 rpc_server.cc:307] RPC server started. Bound to: 127.31.32.62:45713
I20260810 04:09:06.308821 14094 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.32.62:45713 every 8 connection(s)
I20260810 04:09:06.309157 14020 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786334794676589-31872-0/minicluster-data/master-0/data/info.pb
I20260810 04:09:06.310075 14095 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:09:06.311713 14095 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.32.62" port: 45713 } has no permanent_uuid. Determining permanent_uuid...
I20260810 04:09:06.313877 31872 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskws89Io/build/release/bin/kudu as pid 14020
I20260810 04:09:06.313997 31872 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786334794676589-31872-0/minicluster-data/master-0/wal/instance
I20260810 04:09:06.314170 31872 external_mini_cluster.cc:1428] Setting key d6f28fa9752d390783c5dcabc0037667
Aug 10 04:09:06 dist-test-slave-8w9v krb5kdc[14006](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334946, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.62@KRBTEST.COM for kudu/127.31.32.62@KRBTEST.COM
I20260810 04:09:06.321623 14098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:06.312489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:45713 (local address 127.0.0.1:57136)
0810 04:09:06.312694 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0810 04:09:06.312736 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 04:09:06.313256 (+ 520us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:09:06.313586 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:09:06.313589 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:09:06.313880 (+ 291us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:09:06.314068 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:09:06.314075 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:09:06.315132 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:09:06.315133 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:09:06.315798 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:09:06.315800 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:09:06.316384 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:09:06.316385 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:09:06.316448 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:09:06.316968 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:09:06.316977 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:09:06.318287 (+ 1310us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:09:06.320133 (+ 1846us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:09:06.320140 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:09:06.320142 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:09:06.320347 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:09:06.320587 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:09:06.320588 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:09:06.320588 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:09:06.320673 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:09:06.321031 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:09:06.321035 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:09:06.321096 (+ 61us) client_negotiation.cc:769] Sending connection context
0810 04:09:06.321194 (+ 98us) client_negotiation.cc:240] Negotiation successful
0810 04:09:06.321429 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"mutex_wait_us":98,"spinlock_wait_cycles":8448,"thread_start_us":108,"threads_started":1}
I20260810 04:09:06.321625 14097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:06.312488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57136 (local address 127.31.32.62:45713)
0810 04:09:06.312700 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0810 04:09:06.312710 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 04:09:06.312846 (+ 136us) server_negotiation.cc:407] Connection header received
0810 04:09:06.313443 (+ 597us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:09:06.313445 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:09:06.313468 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:09:06.313568 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:09:06.314177 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:09:06.315016 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:09:06.315893 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:09:06.316276 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:09:06.316410 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:09:06.318378 (+ 1968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:09:06.318386 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:09:06.318386 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:09:06.318393 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:09:06.320021 (+ 1628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:09:06.320438 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:09:06.320441 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:09:06.320442 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:09:06.320521 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:09:06.320749 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:09:06.320750 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:09:06.320750 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:09:06.320957 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:09:06.321038 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0810 04:09:06.321219 (+ 181us) server_negotiation.cc:299] Negotiation successful
0810 04:09:06.321380 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":18,"server-negotiator.queue_time_us":166,"thread_start_us":104,"threads_started":1}
WARNING: no policy specified for kudu/127.31.32.61@KRBTEST.COM; defaulting to no policy
I20260810 04:09:06.322518 14095 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.32.61" port: 38055 } has no permanent_uuid. Determining permanent_uuid...
I20260810 04:09:06.322952 14098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:06.322739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.61:38055 (local address 127.0.0.1:35710)
0810 04:09:06.322844 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0810 04:09:06.322901 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.61:38055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
W20260810 04:09:06.323125 14040 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.32.61:38055: connect: Connection refused (error 111)
I20260810 04:09:06.323547 14098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:06.323462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.61:38055 (local address 127.0.0.1:35716)
0810 04:09:06.323504 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0810 04:09:06.323529 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.61:38055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260810 04:09:06.323658 14095 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.32.61:38055: Network error: Client connection negotiation failed: client connection to 127.31.32.61:38055: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.32.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.32.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.61.keytab.
Entry for principal kudu/127.31.32.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.61.keytab.
I20260810 04:09:06.338765 31872 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskws89Io/build/release/bin/kudu
/tmp/dist-test-taskws89Io/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786334794676589-31872-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786334794676589-31872-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786334794676589-31872-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786334794676589-31872-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.32.61:38055
--webserver_interface=127.31.32.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.61.keytab
--principal=kudu/127.31.32.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.32.20:39979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.32.62:45713,127.31.32.61:38055,127.31.32.60:37637
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786334794676589-31872-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786334794676589-31872-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786334794676589-31872-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 04:09:06.372890 14095 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.32.61" port: 38055 } attempt: 1
I20260810 04:09:06.373395 14098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:06.373246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.61:38055 (local address 127.0.0.1:35718)
0810 04:09:06.373333 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0810 04:09:06.373365 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.61:38055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260810 04:09:06.373725 14098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:06.373633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.61:38055 (local address 127.0.0.1:35726)
0810 04:09:06.373677 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0810 04:09:06.373706 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.61:38055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260810 04:09:06.373847 14095 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.32.61:38055: Network error: Client connection negotiation failed: client connection to 127.31.32.61:38055: connect: Connection refused (error 111)
W20260810 04:09:06.419072 14103 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:09:06.419291 14103 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:09:06.419325 14103 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:09:06.420992 14103 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:09:06.421051 14103 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:09:06.421075 14103 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:09:06.421108 14103 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:09:06.421134 14103 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:09:06.422572 14103 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.32.20:39979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786334794676589-31872-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786334794676589-31872-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.31.32.62:45713,127.31.32.61:38055,127.31.32.60:37637
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.61.keytab
--principal=kudu/127.31.32.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.32.61:38055
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786334794676589-31872-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.31.32.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14103
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786334794676589-31872-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:09:06.422838 14103 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:09:06.423063 14103 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:09:06.427728 14111 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:09:06.427757 14109 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:09:06.427724 14108 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:09:06.428242 14103 server_base.cc:1034] running on GCE node
I20260810 04:09:06.428493 14103 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:09:06.428764 14103 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:09:06.429946 14103 hybrid_clock.cc:648] HybridClock initialized: now 1786334946429934 us; error 25 us; skew 500 ppm
Aug 10 04:09:06 dist-test-slave-8w9v krb5kdc[14006](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334946, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:09:06.432415 14103 init.cc:377] Logged in from keytab as kudu/127.31.32.61@KRBTEST.COM (short username kudu)
I20260810 04:09:06.432891 14103 webserver.cc:460] Webserver started at http://127.31.32.61:39399/ using document root <none> and password file <none>
I20260810 04:09:06.433115 14103 fs_manager.cc:359] Metadata directory not provided
I20260810 04:09:06.433177 14103 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:09:06.433321 14103 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:09:06.434345 14103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786334794676589-31872-0/minicluster-data/master-1/data/instance:
uuid: "4e99abddc89a47eb9bc643bf1f7e2522"
format_stamp: "Formatted at 2026-08-10 04:09:06 on dist-test-slave-8w9v"
server_key: "ffe67d9116be640346c4ff4a9154b847"
server_key_iv: "756bd980c5c6d671734140942c1a9217"
server_key_version: "encryptionkey@0"
I20260810 04:09:06.434710 14103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786334794676589-31872-0/minicluster-data/master-1/wal/instance:
uuid: "4e99abddc89a47eb9bc643bf1f7e2522"
format_stamp: "Formatted at 2026-08-10 04:09:06 on dist-test-slave-8w9v"
server_key: "ffe67d9116be640346c4ff4a9154b847"
server_key_iv: "756bd980c5c6d671734140942c1a9217"
server_key_version: "encryptionkey@0"
I20260810 04:09:06.436057 14103 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260810 04:09:06.436767 14119 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:09:06.436944 14103 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 04:09:06.437036 14103 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786334794676589-31872-0/minicluster-data/master-1/data,/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786334794676589-31872-0/minicluster-data/master-1/wal
uuid: "4e99abddc89a47eb9bc643bf1f7e2522"
format_stamp: "Formatted at 2026-08-10 04:09:06 on dist-test-slave-8w9v"
server_key: "ffe67d9116be640346c4ff4a9154b847"
server_key_iv: "756bd980c5c6d671734140942c1a9217"
server_key_version: "encryptionkey@0"
I20260810 04:09:06.437139 14103 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786334794676589-31872-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786334794676589-31872-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786334794676589-31872-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:09:06.442025 14095 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.32.61" port: 38055 } attempt: 2
I20260810 04:09:06.442510 14098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:06.442369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.61:38055 (local address 127.0.0.1:35732)
0810 04:09:06.442435 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0810 04:09:06.442475 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.61:38055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260810 04:09:06.442886 14098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:06.442781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.61:38055 (local address 127.0.0.1:35734)
0810 04:09:06.442828 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0810 04:09:06.442860 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.61:38055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260810 04:09:06.443007 14095 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.32.61:38055: Network error: Client connection negotiation failed: client connection to 127.31.32.61:38055: connect: Connection refused (error 111)
I20260810 04:09:06.446867 14103 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:09:06.449020 14103 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:09:06.449126 14103 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:09:06.452939 14103 rpc_server.cc:307] RPC server started. Bound to: 127.31.32.61:38055
I20260810 04:09:06.452988 14181 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.32.61:38055 every 8 connection(s)
I20260810 04:09:06.453325 14103 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786334794676589-31872-0/minicluster-data/master-1/data/info.pb
I20260810 04:09:06.454121 14182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:09:06.455685 14182 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.32.62" port: 45713 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 04:09:06 dist-test-slave-8w9v krb5kdc[14006](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334946, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.61@KRBTEST.COM for kudu/127.31.32.62@KRBTEST.COM
I20260810 04:09:06.463433 31872 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskws89Io/build/release/bin/kudu as pid 14103
I20260810 04:09:06.463555 31872 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786334794676589-31872-0/minicluster-data/master-1/wal/instance
I20260810 04:09:06.463788 31872 external_mini_cluster.cc:1428] Setting key d5cc57bb3c944e296ceed560bb7e926d
I20260810 04:09:06.464007 14097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:06.456390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57142 (local address 127.31.32.62:45713)
0810 04:09:06.456429 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0810 04:09:06.456431 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:09:06.456636 (+ 205us) server_negotiation.cc:407] Connection header received
0810 04:09:06.457067 (+ 431us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:09:06.457068 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:09:06.457079 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:09:06.457133 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:09:06.457646 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:09:06.458128 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:09:06.459048 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:09:06.459252 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:09:06.459333 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:09:06.461335 (+ 2002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:09:06.461337 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:09:06.461337 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:09:06.461345 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:09:06.462678 (+ 1333us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:09:06.463027 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:09:06.463027 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:09:06.463027 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:09:06.463094 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:09:06.463311 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:09:06.463312 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:09:06.463313 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:09:06.463455 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:09:06.463532 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0810 04:09:06.463810 (+ 278us) server_negotiation.cc:299] Negotiation successful
0810 04:09:06.463898 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260810 04:09:06.464115 14184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:06.456369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:45713 (local address 127.0.0.1:57142)
0810 04:09:06.456519 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0810 04:09:06.456558 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0810 04:09:06.456978 (+ 420us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:09:06.457182 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:09:06.457185 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:09:06.457402 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:09:06.457578 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:09:06.457581 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:09:06.458205 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:09:06.458205 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:09:06.458963 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:09:06.458965 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:09:06.459349 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:09:06.459350 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:09:06.459445 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:09:06.459904 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:09:06.459912 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:09:06.461218 (+ 1306us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:09:06.462746 (+ 1528us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:09:06.462752 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:09:06.462754 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:09:06.462965 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:09:06.463146 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:09:06.463147 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:09:06.463148 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:09:06.463238 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:09:06.463529 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:09:06.463535 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:09:06.463688 (+ 153us) client_negotiation.cc:769] Sending connection context
0810 04:09:06.463771 (+ 83us) client_negotiation.cc:240] Negotiation successful
0810 04:09:06.463936 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":78,"threads_started":1}
I20260810 04:09:06.464730 14182 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.32.61" port: 38055 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 04:09:06 dist-test-slave-8w9v krb5kdc[14006](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334946, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.61@KRBTEST.COM for kudu/127.31.32.61@KRBTEST.COM
I20260810 04:09:06.472290 14184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:06.464957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.61:38055 (local address 127.0.0.1:35750)
0810 04:09:06.465002 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0810 04:09:06.465012 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 04:09:06.465052 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:09:06.465254 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:09:06.465255 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:09:06.465388 (+ 133us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:09:06.465453 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:09:06.465455 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:09:06.466414 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:09:06.466415 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:09:06.466978 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:09:06.466980 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:09:06.467644 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:09:06.467645 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:09:06.467699 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:09:06.468078 (+ 379us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:09:06.468084 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:09:06.469343 (+ 1259us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:09:06.471244 (+ 1901us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:09:06.471245 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:09:06.471246 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:09:06.471414 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:09:06.471666 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:09:06.471667 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:09:06.471667 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:09:06.471717 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:09:06.472036 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:09:06.472037 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:09:06.472066 (+ 29us) client_negotiation.cc:769] Sending connection context
0810 04:09:06.472114 (+ 48us) client_negotiation.cc:240] Negotiation successful
0810 04:09:06.472233 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260810 04:09:06.472520 14186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:06.464997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35750 (local address 127.31.32.61:38055)
0810 04:09:06.465139 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0810 04:09:06.465142 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:09:06.465156 (+ 14us) server_negotiation.cc:407] Connection header received
0810 04:09:06.465171 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:09:06.465172 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:09:06.465182 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:09:06.465254 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:09:06.465548 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:09:06.466331 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:09:06.467056 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:09:06.467569 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:09:06.467650 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:09:06.469433 (+ 1783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:09:06.469441 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:09:06.469442 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:09:06.469448 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:09:06.471165 (+ 1717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:09:06.471503 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:09:06.471506 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:09:06.471507 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:09:06.471599 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:09:06.471777 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:09:06.471778 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:09:06.471779 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:09:06.471959 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:09:06.472060 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0810 04:09:06.472173 (+ 113us) server_negotiation.cc:299] Negotiation successful
0810 04:09:06.472325 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":41,"server-negotiator.queue_time_us":113,"thread_start_us":74,"threads_started":1}
WARNING: no policy specified for kudu/127.31.32.60@KRBTEST.COM; defaulting to no policy
I20260810 04:09:06.473068 14182 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.32.60" port: 37637 } has no permanent_uuid. Determining permanent_uuid...
I20260810 04:09:06.473384 14184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:06.473269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.60:37637 (local address 127.0.0.1:59928)
0810 04:09:06.473311 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0810 04:09:06.473367 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.60:37637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260810 04:09:06.473510 14120 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.32.60:37637: connect: Connection refused (error 111)
I20260810 04:09:06.473949 14184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:06.473826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.60:37637 (local address 127.0.0.1:59940)
0810 04:09:06.473906 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0810 04:09:06.473929 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.60:37637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260810 04:09:06.474049 14182 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.32.60:37637: Network error: Client connection negotiation failed: client connection to 127.31.32.60:37637: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.32.60@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.32.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.60.keytab.
Entry for principal kudu/127.31.32.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.60.keytab.
I20260810 04:09:06.490762 31872 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskws89Io/build/release/bin/kudu
/tmp/dist-test-taskws89Io/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786334794676589-31872-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786334794676589-31872-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786334794676589-31872-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786334794676589-31872-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.32.60:37637
--webserver_interface=127.31.32.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.60.keytab
--principal=kudu/127.31.32.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.32.20:39979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.32.62:45713,127.31.32.61:38055,127.31.32.60:37637
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786334794676589-31872-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786334794676589-31872-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786334794676589-31872-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 04:09:06.523247 14182 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.32.60" port: 37637 } attempt: 1
I20260810 04:09:06.523727 14184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:06.523601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.60:37637 (local address 127.0.0.1:59942)
0810 04:09:06.523660 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0810 04:09:06.523698 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.60:37637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260810 04:09:06.524216 14184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:06.524075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.60:37637 (local address 127.0.0.1:59956)
0810 04:09:06.524159 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0810 04:09:06.524188 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.60:37637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260810 04:09:06.524362 14182 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.32.60:37637: Network error: Client connection negotiation failed: client connection to 127.31.32.60:37637: connect: Connection refused (error 111)
I20260810 04:09:06.534152 14095 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.32.61" port: 38055 } attempt: 3
Aug 10 04:09:06 dist-test-slave-8w9v krb5kdc[14006](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334946, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.62@KRBTEST.COM for kudu/127.31.32.61@KRBTEST.COM
I20260810 04:09:06.541394 14186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:06.534556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35760 (local address 127.31.32.61:38055)
0810 04:09:06.534614 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0810 04:09:06.534616 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:09:06.534629 (+ 13us) server_negotiation.cc:407] Connection header received
0810 04:09:06.534692 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:09:06.534693 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:09:06.534706 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:09:06.534767 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:09:06.535138 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:09:06.535660 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:09:06.536470 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:09:06.536697 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:09:06.536779 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:09:06.538659 (+ 1880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:09:06.538660 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:09:06.538661 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:09:06.538668 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:09:06.540238 (+ 1570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:09:06.540597 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:09:06.540598 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:09:06.540598 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:09:06.540659 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:09:06.540882 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:09:06.540883 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:09:06.540883 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:09:06.540998 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:09:06.541067 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0810 04:09:06.541214 (+ 147us) server_negotiation.cc:299] Negotiation successful
0810 04:09:06.541293 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260810 04:09:06.541395 14098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:06.534501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.61:38055 (local address 127.0.0.1:35760)
0810 04:09:06.534548 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0810 04:09:06.534559 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 04:09:06.534626 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:09:06.534772 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:09:06.534773 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:09:06.534975 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:09:06.535059 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:09:06.535061 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:09:06.535742 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:09:06.535743 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:09:06.536388 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:09:06.536391 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:09:06.536778 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:09:06.536779 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:09:06.536856 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:09:06.537345 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:09:06.537351 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:09:06.538577 (+ 1226us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:09:06.540313 (+ 1736us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:09:06.540315 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:09:06.540315 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:09:06.540498 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:09:06.540730 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:09:06.540730 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:09:06.540731 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:09:06.540790 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:09:06.541068 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:09:06.541069 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:09:06.541119 (+ 50us) client_negotiation.cc:769] Sending connection context
0810 04:09:06.541189 (+ 70us) client_negotiation.cc:240] Negotiation successful
0810 04:09:06.541292 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260810 04:09:06.541769 14095 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.32.60" port: 37637 } has no permanent_uuid. Determining permanent_uuid...
I20260810 04:09:06.542147 14098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:06.542065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.60:37637 (local address 127.0.0.1:59966)
0810 04:09:06.542101 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0810 04:09:06.542130 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.60:37637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260810 04:09:06.542477 14098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:06.542390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.60:37637 (local address 127.0.0.1:59970)
0810 04:09:06.542432 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0810 04:09:06.542456 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.60:37637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260810 04:09:06.542569 14095 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.32.60:37637: Network error: Client connection negotiation failed: client connection to 127.31.32.60:37637: connect: Connection refused (error 111)
I20260810 04:09:06.573697 14095 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.32.60" port: 37637 } attempt: 1
I20260810 04:09:06.574133 14098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:06.574007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.60:37637 (local address 127.0.0.1:59984)
0810 04:09:06.574066 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0810 04:09:06.574104 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.60:37637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260810 04:09:06.574445 14098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:06.574358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.60:37637 (local address 127.0.0.1:59990)
0810 04:09:06.574399 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0810 04:09:06.574427 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.60:37637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260810 04:09:06.574556 14095 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.32.60:37637: Network error: Client connection negotiation failed: client connection to 127.31.32.60:37637: connect: Connection refused (error 111)
W20260810 04:09:06.575513 14190 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:09:06.575714 14190 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:09:06.575749 14190 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:09:06.576921 14190 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:09:06.576982 14190 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:09:06.577008 14190 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:09:06.577055 14190 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:09:06.577082 14190 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:09:06.578497 14190 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.32.20:39979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786334794676589-31872-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786334794676589-31872-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.31.32.62:45713,127.31.32.61:38055,127.31.32.60:37637
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.60.keytab
--principal=kudu/127.31.32.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.32.60:37637
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786334794676589-31872-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.31.32.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14190
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786334794676589-31872-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:09:06.578759 14190 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:09:06.579005 14190 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:09:06.583770 14196 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:09:06.583801 14197 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:09:06.583814 14199 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:09:06.584079 14190 server_base.cc:1034] running on GCE node
I20260810 04:09:06.584339 14190 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:09:06.584671 14190 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:09:06.585832 14190 hybrid_clock.cc:648] HybridClock initialized: now 1786334946585827 us; error 50 us; skew 500 ppm
Aug 10 04:09:06 dist-test-slave-8w9v krb5kdc[14006](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334946, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:09:06.588203 14190 init.cc:377] Logged in from keytab as kudu/127.31.32.60@KRBTEST.COM (short username kudu)
I20260810 04:09:06.588652 14190 webserver.cc:460] Webserver started at http://127.31.32.60:43117/ using document root <none> and password file <none>
I20260810 04:09:06.588886 14190 fs_manager.cc:359] Metadata directory not provided
I20260810 04:09:06.588948 14190 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:09:06.589078 14190 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:09:06.590097 14190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786334794676589-31872-0/minicluster-data/master-2/data/instance:
uuid: "42c1e9b618f746d1baafe3a987c3bd31"
format_stamp: "Formatted at 2026-08-10 04:09:06 on dist-test-slave-8w9v"
server_key: "e129d20bf7d26360e2784061e322ec03"
server_key_iv: "2630f39dc5db20c6358a251cc8e4b811"
server_key_version: "encryptionkey@0"
I20260810 04:09:06.590451 14190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786334794676589-31872-0/minicluster-data/master-2/wal/instance:
uuid: "42c1e9b618f746d1baafe3a987c3bd31"
format_stamp: "Formatted at 2026-08-10 04:09:06 on dist-test-slave-8w9v"
server_key: "e129d20bf7d26360e2784061e322ec03"
server_key_iv: "2630f39dc5db20c6358a251cc8e4b811"
server_key_version: "encryptionkey@0"
I20260810 04:09:06.591849 14190 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 04:09:06.592495 14182 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.32.60" port: 37637 } attempt: 2
I20260810 04:09:06.592744 14209 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:09:06.592877 14184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:06.592732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.60:37637 (local address 127.0.0.1:60000)
0810 04:09:06.592786 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0810 04:09:06.592843 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.60:37637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260810 04:09:06.592904 14190 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260810 04:09:06.592975 14190 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786334794676589-31872-0/minicluster-data/master-2/data,/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786334794676589-31872-0/minicluster-data/master-2/wal
uuid: "42c1e9b618f746d1baafe3a987c3bd31"
format_stamp: "Formatted at 2026-08-10 04:09:06 on dist-test-slave-8w9v"
server_key: "e129d20bf7d26360e2784061e322ec03"
server_key_iv: "2630f39dc5db20c6358a251cc8e4b811"
server_key_version: "encryptionkey@0"
I20260810 04:09:06.593063 14190 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786334794676589-31872-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786334794676589-31872-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786334794676589-31872-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:09:06.593297 14184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:06.593150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.60:37637 (local address 127.0.0.1:60002)
0810 04:09:06.593248 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0810 04:09:06.593276 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.60:37637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
W20260810 04:09:06.593417 14182 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.32.60:37637: Network error: Client connection negotiation failed: client connection to 127.31.32.60:37637: connect: Connection refused (error 111)
I20260810 04:09:06.601557 14190 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:09:06.603253 14190 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:09:06.603407 14190 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:09:06.607348 14190 rpc_server.cc:307] RPC server started. Bound to: 127.31.32.60:37637
I20260810 04:09:06.607407 14264 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.32.60:37637 every 8 connection(s)
I20260810 04:09:06.607728 14190 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786334794676589-31872-0/minicluster-data/master-2/data/info.pb
I20260810 04:09:06.608588 14265 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:09:06.609988 14265 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.32.62" port: 45713 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 04:09:06 dist-test-slave-8w9v krb5kdc[14006](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334946, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.60@KRBTEST.COM for kudu/127.31.32.62@KRBTEST.COM
I20260810 04:09:06.615325 31872 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskws89Io/build/release/bin/kudu as pid 14190
I20260810 04:09:06.615446 31872 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786334794676589-31872-0/minicluster-data/master-2/wal/instance
I20260810 04:09:06.615639 31872 external_mini_cluster.cc:1428] Setting key cb03f821ddf8494ac8526a4bc908c629
I20260810 04:09:06.618775 14097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:06.610614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57146 (local address 127.31.32.62:45713)
0810 04:09:06.610679 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0810 04:09:06.610681 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:09:06.610864 (+ 183us) server_negotiation.cc:407] Connection header received
0810 04:09:06.611284 (+ 420us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:09:06.611285 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:09:06.611294 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:09:06.611361 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:09:06.611912 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:09:06.612422 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:09:06.613402 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:09:06.613589 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:09:06.613641 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:09:06.615733 (+ 2092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:09:06.615735 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:09:06.615735 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:09:06.615743 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:09:06.617482 (+ 1739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:09:06.617852 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:09:06.617853 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:09:06.617853 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:09:06.617913 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:09:06.618171 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:09:06.618172 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:09:06.618172 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:09:06.618277 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:09:06.618335 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0810 04:09:06.618648 (+ 313us) server_negotiation.cc:299] Negotiation successful
0810 04:09:06.618712 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260810 04:09:06.618993 14267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:06.610575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:45713 (local address 127.0.0.1:57146)
0810 04:09:06.610744 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0810 04:09:06.610783 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0810 04:09:06.611206 (+ 423us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:09:06.611422 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:09:06.611425 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:09:06.611653 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:09:06.611831 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:09:06.611835 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:09:06.612507 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:09:06.612507 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:09:06.613328 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:09:06.613331 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:09:06.613657 (+ 326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:09:06.613658 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:09:06.613756 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:09:06.614189 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:09:06.614198 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:09:06.615641 (+ 1443us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:09:06.617569 (+ 1928us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:09:06.617574 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:09:06.617576 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:09:06.617788 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:09:06.617975 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:09:06.617976 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:09:06.617976 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:09:06.618103 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:09:06.618338 (+ 235us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:09:06.618344 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:09:06.618512 (+ 168us) client_negotiation.cc:769] Sending connection context
0810 04:09:06.618627 (+ 115us) client_negotiation.cc:240] Negotiation successful
0810 04:09:06.618775 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":97,"threads_started":1}
I20260810 04:09:06.619567 14265 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.32.61" port: 38055 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 04:09:06 dist-test-slave-8w9v krb5kdc[14006](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334946, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.32.60@KRBTEST.COM
Aug 10 04:09:06 dist-test-slave-8w9v krb5kdc[14006](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334946, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.60@KRBTEST.COM for kudu/127.31.32.61@KRBTEST.COM
I20260810 04:09:06.626085 14269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:06.616019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60016 (local address 127.31.32.60:37637)
0810 04:09:06.616289 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0810 04:09:06.616294 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 04:09:06.616317 (+ 23us) server_negotiation.cc:407] Connection header received
0810 04:09:06.616340 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:09:06.616342 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:09:06.616357 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:09:06.616440 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:09:06.617021 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:09:06.618005 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:09:06.619118 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:09:06.619535 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:09:06.619608 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:09:06.622167 (+ 2559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:09:06.622177 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:09:06.622177 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:09:06.622184 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:09:06.624061 (+ 1877us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:09:06.624518 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:09:06.624522 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:09:06.624522 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:09:06.624590 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:09:06.624814 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:09:06.624815 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:09:06.624815 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:09:06.624990 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:09:06.625164 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0810 04:09:06.625856 (+ 692us) server_negotiation.cc:299] Negotiation successful
0810 04:09:06.625952 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":238,"thread_start_us":78,"threads_started":1}
I20260810 04:09:06.627017 14186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:06.619919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35766 (local address 127.31.32.61:38055)
0810 04:09:06.619967 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0810 04:09:06.619969 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:09:06.619978 (+ 9us) server_negotiation.cc:407] Connection header received
0810 04:09:06.620005 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:09:06.620006 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:09:06.620016 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:09:06.620074 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:09:06.620404 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:09:06.620847 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:09:06.621561 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:09:06.621894 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:09:06.621978 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:09:06.624068 (+ 2090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:09:06.624069 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:09:06.624070 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:09:06.624079 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:09:06.625687 (+ 1608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:09:06.626085 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:09:06.626086 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:09:06.626086 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:09:06.626151 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:09:06.626470 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:09:06.626471 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:09:06.626471 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:09:06.626609 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:09:06.626697 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0810 04:09:06.626871 (+ 174us) server_negotiation.cc:299] Negotiation successful
0810 04:09:06.626955 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260810 04:09:06.627023 14267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:06.619816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.61:38055 (local address 127.0.0.1:35766)
0810 04:09:06.619859 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0810 04:09:06.619867 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0810 04:09:06.619902 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:09:06.620091 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:09:06.620092 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:09:06.620264 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:09:06.620316 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:09:06.620318 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:09:06.620922 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:09:06.620923 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:09:06.621465 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:09:06.621467 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:09:06.621985 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:09:06.621986 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:09:06.622037 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:09:06.622426 (+ 389us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:09:06.622431 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:09:06.623974 (+ 1543us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:09:06.625762 (+ 1788us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:09:06.625763 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:09:06.625763 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:09:06.626014 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:09:06.626224 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:09:06.626225 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:09:06.626225 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:09:06.626369 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:09:06.626699 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:09:06.626700 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:09:06.626755 (+ 55us) client_negotiation.cc:769] Sending connection context
0810 04:09:06.626825 (+ 70us) client_negotiation.cc:240] Negotiation successful
0810 04:09:06.626914 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260810 04:09:06.627393 14265 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.32.60" port: 37637 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 04:09:06 dist-test-slave-8w9v krb5kdc[14006](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334946, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.60@KRBTEST.COM for kudu/127.31.32.60@KRBTEST.COM
Aug 10 04:09:06 dist-test-slave-8w9v krb5kdc[14006](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334946, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.32.62@KRBTEST.COM
I20260810 04:09:06.634155 14267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:06.627633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.60:37637 (local address 127.0.0.1:60020)
0810 04:09:06.627722 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0810 04:09:06.627732 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 04:09:06.627778 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:09:06.627907 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:09:06.627908 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:09:06.628039 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:09:06.628062 (+ 23us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:09:06.628063 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:09:06.628696 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:09:06.628697 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:09:06.629134 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:09:06.629135 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:09:06.629644 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:09:06.629646 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:09:06.629710 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:09:06.630005 (+ 295us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:09:06.630010 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:09:06.631260 (+ 1250us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:09:06.633180 (+ 1920us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:09:06.633181 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:09:06.633181 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:09:06.633392 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:09:06.633582 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:09:06.633583 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:09:06.633583 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:09:06.633647 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:09:06.633848 (+ 201us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:09:06.633848 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:09:06.633905 (+ 57us) client_negotiation.cc:769] Sending connection context
0810 04:09:06.633984 (+ 79us) client_negotiation.cc:240] Negotiation successful
0810 04:09:06.634080 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":78}
I20260810 04:09:06.634337 14269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:06.627688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60020 (local address 127.31.32.60:37637)
0810 04:09:06.627721 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0810 04:09:06.627722 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 04:09:06.627764 (+ 42us) server_negotiation.cc:407] Connection header received
0810 04:09:06.627836 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:09:06.627837 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:09:06.627847 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:09:06.627904 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:09:06.628155 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:09:06.628603 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:09:06.629250 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:09:06.629552 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:09:06.629620 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:09:06.631348 (+ 1728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:09:06.631349 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:09:06.631350 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:09:06.631357 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:09:06.633087 (+ 1730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:09:06.633463 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:09:06.633464 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:09:06.633464 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:09:06.633517 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:09:06.633703 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:09:06.633703 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:09:06.633703 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:09:06.633799 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:09:06.633850 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0810 04:09:06.634000 (+ 150us) server_negotiation.cc:299] Negotiation successful
0810 04:09:06.634172 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":85,"server-negotiator.queue_time_us":16}
I20260810 04:09:06.635166 14097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:06.626907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57150 (local address 127.31.32.62:45713)
0810 04:09:06.626944 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0810 04:09:06.626946 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:09:06.627013 (+ 67us) server_negotiation.cc:407] Connection header received
0810 04:09:06.627107 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:09:06.627108 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:09:06.627129 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:09:06.627157 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:09:06.627763 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:09:06.628265 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:09:06.628909 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:09:06.629103 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:09:06.629170 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:09:06.632043 (+ 2873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:09:06.632044 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:09:06.632045 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:09:06.632052 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:09:06.633732 (+ 1680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:09:06.634066 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:09:06.634067 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:09:06.634067 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:09:06.634149 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:09:06.634570 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:09:06.634571 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:09:06.634572 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:09:06.634722 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:09:06.634927 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0810 04:09:06.634969 (+ 42us) server_negotiation.cc:299] Negotiation successful
0810 04:09:06.635045 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260810 04:09:06.636112 14265 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42c1e9b618f746d1baafe3a987c3bd31: Bootstrap starting.
I20260810 04:09:06.637164 14265 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 42c1e9b618f746d1baafe3a987c3bd31: Neither blocks nor log segments found. Creating new log.
I20260810 04:09:06.637477 14265 log.cc:824] T 00000000000000000000000000000000 P 42c1e9b618f746d1baafe3a987c3bd31: Log is configured to *not* fsync() on all Append() calls
I20260810 04:09:06.638208 14265 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42c1e9b618f746d1baafe3a987c3bd31: No bootstrap required, opened a new log
I20260810 04:09:06.639899 14265 raft_consensus.cc:348] T 00000000000000000000000000000000 P 42c1e9b618f746d1baafe3a987c3bd31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44c973ce2d7748e0b843e447666bc8ca" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 45713 } } peers { permanent_uuid: "4e99abddc89a47eb9bc643bf1f7e2522" member_type: VOTER last_known_addr { host: "127.31.32.61" port: 38055 } } peers { permanent_uuid: "42c1e9b618f746d1baafe3a987c3bd31" member_type: VOTER last_known_addr { host: "127.31.32.60" port: 37637 } }
I20260810 04:09:06.640039 14265 raft_consensus.cc:374] T 00000000000000000000000000000000 P 42c1e9b618f746d1baafe3a987c3bd31 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
Aug 10 04:09:06 dist-test-slave-8w9v krb5kdc[14006](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334946, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.32.61@KRBTEST.COM
I20260810 04:09:06.640087 14265 raft_consensus.cc:729] T 00000000000000000000000000000000 P 42c1e9b618f746d1baafe3a987c3bd31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42c1e9b618f746d1baafe3a987c3bd31, State: Initialized, Role: FOLLOWER
I20260810 04:09:06.640277 14265 consensus_queue.cc:260] T 00000000000000000000000000000000 P 42c1e9b618f746d1baafe3a987c3bd31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44c973ce2d7748e0b843e447666bc8ca" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 45713 } } peers { permanent_uuid: "4e99abddc89a47eb9bc643bf1f7e2522" member_type: VOTER last_known_addr { host: "127.31.32.61" port: 38055 } } peers { permanent_uuid: "42c1e9b618f746d1baafe3a987c3bd31" member_type: VOTER last_known_addr { host: "127.31.32.60" port: 37637 } }
I20260810 04:09:06.640581 14271 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42c1e9b618f746d1baafe3a987c3bd31 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44c973ce2d7748e0b843e447666bc8ca" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 45713 } } peers { permanent_uuid: "4e99abddc89a47eb9bc643bf1f7e2522" member_type: VOTER last_known_addr { host: "127.31.32.61" port: 38055 } } peers { permanent_uuid: "42c1e9b618f746d1baafe3a987c3bd31" member_type: VOTER last_known_addr { host: "127.31.32.60" port: 37637 } } }
I20260810 04:09:06.640622 14265 sys_catalog.cc:564] T 00000000000000000000000000000000 P 42c1e9b618f746d1baafe3a987c3bd31 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:09:06.640687 14271 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42c1e9b618f746d1baafe3a987c3bd31 [sys.catalog]: This master's current role is: FOLLOWER
W20260810 04:09:06.642676 14283 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 42c1e9b618f746d1baafe3a987c3bd31: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 04:09:06.642740 14283 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 04:09:06.643637 14186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:06.635917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35768 (local address 127.31.32.61:38055)
0810 04:09:06.635954 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0810 04:09:06.635956 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:09:06.636000 (+ 44us) server_negotiation.cc:407] Connection header received
0810 04:09:06.636021 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:09:06.636022 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:09:06.636031 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:09:06.636080 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:09:06.636631 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:09:06.637335 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:09:06.637998 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:09:06.638201 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:09:06.638279 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:09:06.640698 (+ 2419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:09:06.640700 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:09:06.640700 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:09:06.640708 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:09:06.642414 (+ 1706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:09:06.642816 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:09:06.642817 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:09:06.642817 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:09:06.642877 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:09:06.643108 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:09:06.643109 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:09:06.643110 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:09:06.643242 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:09:06.643433 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0810 04:09:06.643474 (+ 41us) server_negotiation.cc:299] Negotiation successful
0810 04:09:06.643551 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260810 04:09:06.642988 14190 master_runner.cc:186] Error getting master registration for 127.31.32.62:45713: OK, instance_id { permanent_uuid: "44c973ce2d7748e0b843e447666bc8ca" instance_seqno: 1786334946307303 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 04:09:06.645226 14190 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 04:09:06.645505 31872 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 04:09:06.649698 14095 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.32.60" port: 37637 } attempt: 2
I20260810 04:09:06.654946 14186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:06.646870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35780 (local address 127.31.32.61:38055)
0810 04:09:06.646926 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0810 04:09:06.646927 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 04:09:06.647214 (+ 287us) server_negotiation.cc:407] Connection header received
0810 04:09:06.647313 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:09:06.647314 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:09:06.647323 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:09:06.647372 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:09:06.648114 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:09:06.648841 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:09:06.650474 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:09:06.650741 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:09:06.650799 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:09:06.652241 (+ 1442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:09:06.652242 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:09:06.652242 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:09:06.652249 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:09:06.653742 (+ 1493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:09:06.654138 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:09:06.654140 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:09:06.654140 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:09:06.654192 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:09:06.654385 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:09:06.654386 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:09:06.654387 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:09:06.654501 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:09:06.654581 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0810 04:09:06.654776 (+ 195us) server_negotiation.cc:299] Negotiation successful
0810 04:09:06.654862 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
Aug 10 04:09:06 dist-test-slave-8w9v krb5kdc[14006](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334946, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.62@KRBTEST.COM for kudu/127.31.32.60@KRBTEST.COM
I20260810 04:09:06.654997 14269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:06.647039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60032 (local address 127.31.32.60:37637)
0810 04:09:06.647076 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0810 04:09:06.647078 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:09:06.647406 (+ 328us) server_negotiation.cc:407] Connection header received
0810 04:09:06.647503 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:09:06.647504 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:09:06.647514 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:09:06.647542 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:09:06.648277 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:09:06.648970 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:09:06.649890 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:09:06.650159 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:09:06.650193 (+ 34us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:09:06.651966 (+ 1773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:09:06.651968 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:09:06.651968 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:09:06.651975 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:09:06.653548 (+ 1573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:09:06.653935 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:09:06.653936 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:09:06.653936 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:09:06.653993 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:09:06.654176 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:09:06.654177 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:09:06.654177 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:09:06.654337 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:09:06.654453 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0810 04:09:06.654841 (+ 388us) server_negotiation.cc:299] Negotiation successful
0810 04:09:06.654914 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260810 04:09:06.658725 14097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:06.646927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57156 (local address 127.31.32.62:45713)
0810 04:09:06.646963 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0810 04:09:06.646965 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:09:06.647189 (+ 224us) server_negotiation.cc:407] Connection header received
0810 04:09:06.647265 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:09:06.647266 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:09:06.647276 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:09:06.647327 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:09:06.648168 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:09:06.648890 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:09:06.649857 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:09:06.650143 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:09:06.650184 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:09:06.655940 (+ 5756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:09:06.655942 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:09:06.655942 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:09:06.655948 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:09:06.657389 (+ 1441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:09:06.657797 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:09:06.657798 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:09:06.657798 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:09:06.657869 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:09:06.658094 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:09:06.658095 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:09:06.658096 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:09:06.658219 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:09:06.658307 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0810 04:09:06.658532 (+ 225us) server_negotiation.cc:299] Negotiation successful
0810 04:09:06.658608 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260810 04:09:06.659435 14098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:06.649930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.60:37637 (local address 127.0.0.1:60046)
0810 04:09:06.650053 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0810 04:09:06.650064 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 04:09:06.650102 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:09:06.650701 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:09:06.650702 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:09:06.650923 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:09:06.650962 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:09:06.650964 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:09:06.651999 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:09:06.652001 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:09:06.652790 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:09:06.652794 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:09:06.653375 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:09:06.653376 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:09:06.653435 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:09:06.653795 (+ 360us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:09:06.653801 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:09:06.655331 (+ 1530us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:09:06.658289 (+ 2958us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:09:06.658290 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:09:06.658291 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:09:06.658511 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:09:06.658723 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:09:06.658724 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:09:06.658724 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:09:06.658838 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:09:06.659222 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:09:06.659223 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:09:06.659258 (+ 35us) client_negotiation.cc:769] Sending connection context
0810 04:09:06.659310 (+ 52us) client_negotiation.cc:240] Negotiation successful
0810 04:09:06.659377 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107}
I20260810 04:09:06.659627 14292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:06.650192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60046 (local address 127.31.32.60:37637)
0810 04:09:06.650341 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0810 04:09:06.650342 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 04:09:06.650577 (+ 235us) server_negotiation.cc:407] Connection header received
0810 04:09:06.650600 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:09:06.650601 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:09:06.650610 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:09:06.650723 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:09:06.651052 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:09:06.651911 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:09:06.652901 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:09:06.653277 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:09:06.653572 (+ 295us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:09:06.655424 (+ 1852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:09:06.655426 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:09:06.655426 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:09:06.655433 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:09:06.658220 (+ 2787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:09:06.658594 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:09:06.658595 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:09:06.658595 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:09:06.658651 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:09:06.658919 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:09:06.658920 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:09:06.658921 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:09:06.659157 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:09:06.659226 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0810 04:09:06.659333 (+ 107us) server_negotiation.cc:299] Negotiation successful
0810 04:09:06.659412 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":95,"threads_started":1}
I20260810 04:09:06.661677 14095 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44c973ce2d7748e0b843e447666bc8ca: Bootstrap starting.
I20260810 04:09:06.662405 14095 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 44c973ce2d7748e0b843e447666bc8ca: Neither blocks nor log segments found. Creating new log.
I20260810 04:09:06.662714 14095 log.cc:824] T 00000000000000000000000000000000 P 44c973ce2d7748e0b843e447666bc8ca: Log is configured to *not* fsync() on all Append() calls
I20260810 04:09:06.663525 14095 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44c973ce2d7748e0b843e447666bc8ca: No bootstrap required, opened a new log
I20260810 04:09:06.665863 14095 raft_consensus.cc:348] T 00000000000000000000000000000000 P 44c973ce2d7748e0b843e447666bc8ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44c973ce2d7748e0b843e447666bc8ca" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 45713 } } peers { permanent_uuid: "4e99abddc89a47eb9bc643bf1f7e2522" member_type: VOTER last_known_addr { host: "127.31.32.61" port: 38055 } } peers { permanent_uuid: "42c1e9b618f746d1baafe3a987c3bd31" member_type: VOTER last_known_addr { host: "127.31.32.60" port: 37637 } }
I20260810 04:09:06.666002 14095 raft_consensus.cc:374] T 00000000000000000000000000000000 P 44c973ce2d7748e0b843e447666bc8ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:09:06.666036 14095 raft_consensus.cc:729] T 00000000000000000000000000000000 P 44c973ce2d7748e0b843e447666bc8ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44c973ce2d7748e0b843e447666bc8ca, State: Initialized, Role: FOLLOWER
I20260810 04:09:06.666177 14095 consensus_queue.cc:260] T 00000000000000000000000000000000 P 44c973ce2d7748e0b843e447666bc8ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44c973ce2d7748e0b843e447666bc8ca" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 45713 } } peers { permanent_uuid: "4e99abddc89a47eb9bc643bf1f7e2522" member_type: VOTER last_known_addr { host: "127.31.32.61" port: 38055 } } peers { permanent_uuid: "42c1e9b618f746d1baafe3a987c3bd31" member_type: VOTER last_known_addr { host: "127.31.32.60" port: 37637 } }
I20260810 04:09:06.666422 14295 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44c973ce2d7748e0b843e447666bc8ca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44c973ce2d7748e0b843e447666bc8ca" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 45713 } } peers { permanent_uuid: "4e99abddc89a47eb9bc643bf1f7e2522" member_type: VOTER last_known_addr { host: "127.31.32.61" port: 38055 } } peers { permanent_uuid: "42c1e9b618f746d1baafe3a987c3bd31" member_type: VOTER last_known_addr { host: "127.31.32.60" port: 37637 } } }
I20260810 04:09:06.666481 14095 sys_catalog.cc:564] T 00000000000000000000000000000000 P 44c973ce2d7748e0b843e447666bc8ca [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:09:06.666529 14295 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44c973ce2d7748e0b843e447666bc8ca [sys.catalog]: This master's current role is: FOLLOWER
W20260810 04:09:06.668404 14306 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 44c973ce2d7748e0b843e447666bc8ca: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 04:09:06.668457 14306 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 04:09:06.669492 14020 master_runner.cc:186] Error getting master registration for 127.31.32.61:38055: OK, instance_id { permanent_uuid: "4e99abddc89a47eb9bc643bf1f7e2522" instance_seqno: 1786334946451835 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 04:09:06.671025 14020 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 04:09:06.684568 14182 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.32.60" port: 37637 } attempt: 3
Aug 10 04:09:06 dist-test-slave-8w9v krb5kdc[14006](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334946, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.61@KRBTEST.COM for kudu/127.31.32.60@KRBTEST.COM
I20260810 04:09:06.691690 14292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:06.684920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60062 (local address 127.31.32.60:37637)
0810 04:09:06.684959 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0810 04:09:06.684961 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:09:06.684970 (+ 9us) server_negotiation.cc:407] Connection header received
0810 04:09:06.685037 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:09:06.685038 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:09:06.685048 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:09:06.685110 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:09:06.685458 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:09:06.686124 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:09:06.686788 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:09:06.687003 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:09:06.687076 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:09:06.689105 (+ 2029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:09:06.689106 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:09:06.689107 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:09:06.689114 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:09:06.690570 (+ 1456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:09:06.690935 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:09:06.690937 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:09:06.690937 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:09:06.690993 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:09:06.691184 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:09:06.691185 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:09:06.691185 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:09:06.691301 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:09:06.691373 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0810 04:09:06.691517 (+ 144us) server_negotiation.cc:299] Negotiation successful
0810 04:09:06.691593 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260810 04:09:06.691690 14184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:06.684874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.60:37637 (local address 127.0.0.1:60062)
0810 04:09:06.684926 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0810 04:09:06.684938 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 04:09:06.684982 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:09:06.685114 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:09:06.685115 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:09:06.685288 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:09:06.685330 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:09:06.685332 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:09:06.686203 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:09:06.686204 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:09:06.686709 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:09:06.686710 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:09:06.687072 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:09:06.687073 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:09:06.687127 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:09:06.687694 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:09:06.687699 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:09:06.689023 (+ 1324us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:09:06.690671 (+ 1648us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:09:06.690672 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:09:06.690672 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:09:06.690846 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:09:06.691061 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:09:06.691062 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:09:06.691062 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:09:06.691123 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:09:06.691365 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:09:06.691366 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:09:06.691412 (+ 46us) client_negotiation.cc:769] Sending connection context
0810 04:09:06.691492 (+ 80us) client_negotiation.cc:240] Negotiation successful
0810 04:09:06.691593 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260810 04:09:06.693121 14182 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e99abddc89a47eb9bc643bf1f7e2522: Bootstrap starting.
I20260810 04:09:06.693770 14182 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4e99abddc89a47eb9bc643bf1f7e2522: Neither blocks nor log segments found. Creating new log.
I20260810 04:09:06.694083 14182 log.cc:824] T 00000000000000000000000000000000 P 4e99abddc89a47eb9bc643bf1f7e2522: Log is configured to *not* fsync() on all Append() calls
I20260810 04:09:06.695266 14182 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e99abddc89a47eb9bc643bf1f7e2522: No bootstrap required, opened a new log
I20260810 04:09:06.696964 14182 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4e99abddc89a47eb9bc643bf1f7e2522 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44c973ce2d7748e0b843e447666bc8ca" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 45713 } } peers { permanent_uuid: "4e99abddc89a47eb9bc643bf1f7e2522" member_type: VOTER last_known_addr { host: "127.31.32.61" port: 38055 } } peers { permanent_uuid: "42c1e9b618f746d1baafe3a987c3bd31" member_type: VOTER last_known_addr { host: "127.31.32.60" port: 37637 } }
I20260810 04:09:06.697093 14182 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4e99abddc89a47eb9bc643bf1f7e2522 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:09:06.697139 14182 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4e99abddc89a47eb9bc643bf1f7e2522 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e99abddc89a47eb9bc643bf1f7e2522, State: Initialized, Role: FOLLOWER
I20260810 04:09:06.697283 14182 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4e99abddc89a47eb9bc643bf1f7e2522 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44c973ce2d7748e0b843e447666bc8ca" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 45713 } } peers { permanent_uuid: "4e99abddc89a47eb9bc643bf1f7e2522" member_type: VOTER last_known_addr { host: "127.31.32.61" port: 38055 } } peers { permanent_uuid: "42c1e9b618f746d1baafe3a987c3bd31" member_type: VOTER last_known_addr { host: "127.31.32.60" port: 37637 } }
I20260810 04:09:06.697544 14309 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e99abddc89a47eb9bc643bf1f7e2522 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44c973ce2d7748e0b843e447666bc8ca" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 45713 } } peers { permanent_uuid: "4e99abddc89a47eb9bc643bf1f7e2522" member_type: VOTER last_known_addr { host: "127.31.32.61" port: 38055 } } peers { permanent_uuid: "42c1e9b618f746d1baafe3a987c3bd31" member_type: VOTER last_known_addr { host: "127.31.32.60" port: 37637 } } }
I20260810 04:09:06.697609 14182 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4e99abddc89a47eb9bc643bf1f7e2522 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:09:06.697678 14309 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e99abddc89a47eb9bc643bf1f7e2522 [sys.catalog]: This master's current role is: FOLLOWER
W20260810 04:09:06.699371 14322 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4e99abddc89a47eb9bc643bf1f7e2522: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 04:09:06.699474 14322 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260810 04:09:06.699620 14125 tablet.cc:2369] T 00000000000000000000000000000000 P 4e99abddc89a47eb9bc643bf1f7e2522: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 04:09:06.701702 14103 master_runner.cc:263] No leader master found from master 4e99abddc89a47eb9bc643bf1f7e2522
I20260810 04:09:06.701749 14103 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 04:09:06.777833 14309 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4e99abddc89a47eb9bc643bf1f7e2522 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 04:09:06.777989 14309 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4e99abddc89a47eb9bc643bf1f7e2522 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44c973ce2d7748e0b843e447666bc8ca" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 45713 } } peers { permanent_uuid: "4e99abddc89a47eb9bc643bf1f7e2522" member_type: VOTER last_known_addr { host: "127.31.32.61" port: 38055 } } peers { permanent_uuid: "42c1e9b618f746d1baafe3a987c3bd31" member_type: VOTER last_known_addr { host: "127.31.32.60" port: 37637 } }
I20260810 04:09:06.778260 14309 leader_election.cc:288] T 00000000000000000000000000000000 P 4e99abddc89a47eb9bc643bf1f7e2522 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44c973ce2d7748e0b843e447666bc8ca (127.31.32.62:45713), 42c1e9b618f746d1baafe3a987c3bd31 (127.31.32.60:37637)
I20260810 04:09:06.778786 14238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4e99abddc89a47eb9bc643bf1f7e2522" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42c1e9b618f746d1baafe3a987c3bd31" is_pre_election: true
I20260810 04:09:06.778963 14238 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 42c1e9b618f746d1baafe3a987c3bd31 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e99abddc89a47eb9bc643bf1f7e2522 in term 0.
I20260810 04:09:06.779335 14120 leader_election.cc:302] T 00000000000000000000000000000000 P 4e99abddc89a47eb9bc643bf1f7e2522 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42c1e9b618f746d1baafe3a987c3bd31, 4e99abddc89a47eb9bc643bf1f7e2522; no voters:
I20260810 04:09:06.779649 14067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4e99abddc89a47eb9bc643bf1f7e2522" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44c973ce2d7748e0b843e447666bc8ca" is_pre_election: true
I20260810 04:09:06.779783 14067 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 44c973ce2d7748e0b843e447666bc8ca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e99abddc89a47eb9bc643bf1f7e2522 in term 0.
I20260810 04:09:06.779959 14309 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4e99abddc89a47eb9bc643bf1f7e2522 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 04:09:06.780045 14309 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4e99abddc89a47eb9bc643bf1f7e2522 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 04:09:06.780073 14309 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4e99abddc89a47eb9bc643bf1f7e2522 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:09:06.781147 14309 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4e99abddc89a47eb9bc643bf1f7e2522 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44c973ce2d7748e0b843e447666bc8ca" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 45713 } } peers { permanent_uuid: "4e99abddc89a47eb9bc643bf1f7e2522" member_type: VOTER last_known_addr { host: "127.31.32.61" port: 38055 } } peers { permanent_uuid: "42c1e9b618f746d1baafe3a987c3bd31" member_type: VOTER last_known_addr { host: "127.31.32.60" port: 37637 } }
I20260810 04:09:06.781272 14309 leader_election.cc:288] T 00000000000000000000000000000000 P 4e99abddc89a47eb9bc643bf1f7e2522 [CANDIDATE]: Term 1 election: Requested vote from peers 44c973ce2d7748e0b843e447666bc8ca (127.31.32.62:45713), 42c1e9b618f746d1baafe3a987c3bd31 (127.31.32.60:37637)
I20260810 04:09:06.781805 14067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4e99abddc89a47eb9bc643bf1f7e2522" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44c973ce2d7748e0b843e447666bc8ca"
I20260810 04:09:06.781886 14067 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 44c973ce2d7748e0b843e447666bc8ca [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:09:06.782779 14067 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 44c973ce2d7748e0b843e447666bc8ca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e99abddc89a47eb9bc643bf1f7e2522 in term 1.
I20260810 04:09:06.783097 14121 leader_election.cc:302] T 00000000000000000000000000000000 P 4e99abddc89a47eb9bc643bf1f7e2522 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44c973ce2d7748e0b843e447666bc8ca, 4e99abddc89a47eb9bc643bf1f7e2522; no voters:
I20260810 04:09:06.783233 14238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4e99abddc89a47eb9bc643bf1f7e2522" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42c1e9b618f746d1baafe3a987c3bd31"
I20260810 04:09:06.783309 14238 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 42c1e9b618f746d1baafe3a987c3bd31 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:09:06.784236 14238 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 42c1e9b618f746d1baafe3a987c3bd31 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e99abddc89a47eb9bc643bf1f7e2522 in term 1.
I20260810 04:09:06.784540 14309 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4e99abddc89a47eb9bc643bf1f7e2522 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:09:06.784732 14309 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4e99abddc89a47eb9bc643bf1f7e2522 [term 1 LEADER]: Becoming Leader. State: Replica: 4e99abddc89a47eb9bc643bf1f7e2522, State: Running, Role: LEADER
I20260810 04:09:06.784819 14309 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4e99abddc89a47eb9bc643bf1f7e2522 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44c973ce2d7748e0b843e447666bc8ca" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 45713 } } peers { permanent_uuid: "4e99abddc89a47eb9bc643bf1f7e2522" member_type: VOTER last_known_addr { host: "127.31.32.61" port: 38055 } } peers { permanent_uuid: "42c1e9b618f746d1baafe3a987c3bd31" member_type: VOTER last_known_addr { host: "127.31.32.60" port: 37637 } }
I20260810 04:09:06.785207 14309 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e99abddc89a47eb9bc643bf1f7e2522 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4e99abddc89a47eb9bc643bf1f7e2522. Latest consensus state: current_term: 1 leader_uuid: "4e99abddc89a47eb9bc643bf1f7e2522" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44c973ce2d7748e0b843e447666bc8ca" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 45713 } } peers { permanent_uuid: "4e99abddc89a47eb9bc643bf1f7e2522" member_type: VOTER last_known_addr { host: "127.31.32.61" port: 38055 } } peers { permanent_uuid: "42c1e9b618f746d1baafe3a987c3bd31" member_type: VOTER last_known_addr { host: "127.31.32.60" port: 37637 } } }
I20260810 04:09:06.785298 14309 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e99abddc89a47eb9bc643bf1f7e2522 [sys.catalog]: This master's current role is: LEADER
I20260810 04:09:06.786274 14326 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 04:09:06.786499 14326 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 04:09:06.788046 14238 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 42c1e9b618f746d1baafe3a987c3bd31 [term 1 FOLLOWER]: Refusing update from remote peer 4e99abddc89a47eb9bc643bf1f7e2522: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:09:06.788455 14309 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4e99abddc89a47eb9bc643bf1f7e2522 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42c1e9b618f746d1baafe3a987c3bd31" member_type: VOTER last_known_addr { host: "127.31.32.60" port: 37637 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:09:06.789829 14271 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42c1e9b618f746d1baafe3a987c3bd31 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4e99abddc89a47eb9bc643bf1f7e2522. Latest consensus state: current_term: 1 leader_uuid: "4e99abddc89a47eb9bc643bf1f7e2522" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44c973ce2d7748e0b843e447666bc8ca" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 45713 } } peers { permanent_uuid: "4e99abddc89a47eb9bc643bf1f7e2522" member_type: VOTER last_known_addr { host: "127.31.32.61" port: 38055 } } peers { permanent_uuid: "42c1e9b618f746d1baafe3a987c3bd31" member_type: VOTER last_known_addr { host: "127.31.32.60" port: 37637 } } }
I20260810 04:09:06.789944 14271 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42c1e9b618f746d1baafe3a987c3bd31 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 04:09:06.790378 14067 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 44c973ce2d7748e0b843e447666bc8ca [term 1 FOLLOWER]: Refusing update from remote peer 4e99abddc89a47eb9bc643bf1f7e2522: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:09:06.790666 14309 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4e99abddc89a47eb9bc643bf1f7e2522 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44c973ce2d7748e0b843e447666bc8ca" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 45713 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:09:06.791872 14295 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44c973ce2d7748e0b843e447666bc8ca [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4e99abddc89a47eb9bc643bf1f7e2522. Latest consensus state: current_term: 1 leader_uuid: "4e99abddc89a47eb9bc643bf1f7e2522" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44c973ce2d7748e0b843e447666bc8ca" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 45713 } } peers { permanent_uuid: "4e99abddc89a47eb9bc643bf1f7e2522" member_type: VOTER last_known_addr { host: "127.31.32.61" port: 38055 } } peers { permanent_uuid: "42c1e9b618f746d1baafe3a987c3bd31" member_type: VOTER last_known_addr { host: "127.31.32.60" port: 37637 } } }
I20260810 04:09:06.792004 14295 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44c973ce2d7748e0b843e447666bc8ca [sys.catalog]: This master's current role is: FOLLOWER
I20260810 04:09:06.793025 14324 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e99abddc89a47eb9bc643bf1f7e2522 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4e99abddc89a47eb9bc643bf1f7e2522" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44c973ce2d7748e0b843e447666bc8ca" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 45713 } } peers { permanent_uuid: "4e99abddc89a47eb9bc643bf1f7e2522" member_type: VOTER last_known_addr { host: "127.31.32.61" port: 38055 } } peers { permanent_uuid: "42c1e9b618f746d1baafe3a987c3bd31" member_type: VOTER last_known_addr { host: "127.31.32.60" port: 37637 } } }
I20260810 04:09:06.793102 14324 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e99abddc89a47eb9bc643bf1f7e2522 [sys.catalog]: This master's current role is: LEADER
I20260810 04:09:06.794412 14295 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44c973ce2d7748e0b843e447666bc8ca [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4e99abddc89a47eb9bc643bf1f7e2522" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44c973ce2d7748e0b843e447666bc8ca" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 45713 } } peers { permanent_uuid: "4e99abddc89a47eb9bc643bf1f7e2522" member_type: VOTER last_known_addr { host: "127.31.32.61" port: 38055 } } peers { permanent_uuid: "42c1e9b618f746d1baafe3a987c3bd31" member_type: VOTER last_known_addr { host: "127.31.32.60" port: 37637 } } }
I20260810 04:09:06.794498 14295 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44c973ce2d7748e0b843e447666bc8ca [sys.catalog]: This master's current role is: FOLLOWER
I20260810 04:09:06.794783 14327 mvcc.cc:205] Tried to move back new op lower bound from 7316827942041329664 to 7316827942031118336. Current Snapshot: MvccSnapshot[applied={T|T < 7316827942041329664}]
I20260810 04:09:06.795429 14330 mvcc.cc:205] Tried to move back new op lower bound from 7316827942041329664 to 7316827942031118336. Current Snapshot: MvccSnapshot[applied={T|T < 7316827942041329664}]
I20260810 04:09:06.797446 14326 catalog_manager.cc:1347] Generated new cluster ID: 7b07db4c07f04675837200d59fa7f42a
I20260810 04:09:06.797492 14326 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 04:09:06.808593 14309 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e99abddc89a47eb9bc643bf1f7e2522 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4e99abddc89a47eb9bc643bf1f7e2522" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44c973ce2d7748e0b843e447666bc8ca" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 45713 } } peers { permanent_uuid: "4e99abddc89a47eb9bc643bf1f7e2522" member_type: VOTER last_known_addr { host: "127.31.32.61" port: 38055 } } peers { permanent_uuid: "42c1e9b618f746d1baafe3a987c3bd31" member_type: VOTER last_known_addr { host: "127.31.32.60" port: 37637 } } }
I20260810 04:09:06.808715 14309 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e99abddc89a47eb9bc643bf1f7e2522 [sys.catalog]: This master's current role is: LEADER
I20260810 04:09:06.809376 14271 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42c1e9b618f746d1baafe3a987c3bd31 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4e99abddc89a47eb9bc643bf1f7e2522" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44c973ce2d7748e0b843e447666bc8ca" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 45713 } } peers { permanent_uuid: "4e99abddc89a47eb9bc643bf1f7e2522" member_type: VOTER last_known_addr { host: "127.31.32.61" port: 38055 } } peers { permanent_uuid: "42c1e9b618f746d1baafe3a987c3bd31" member_type: VOTER last_known_addr { host: "127.31.32.60" port: 37637 } } }
I20260810 04:09:06.809478 14271 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42c1e9b618f746d1baafe3a987c3bd31 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 04:09:06.809793 14328 mvcc.cc:205] Tried to move back new op lower bound from 7316827942041329664 to 7316827942031118336. Current Snapshot: MvccSnapshot[applied={T|T < 7316827942041329664}]
I20260810 04:09:06.819316 14326 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 04:09:06.820101 14326 catalog_manager.cc:1504] Loading token signing keys...
I20260810 04:09:06.828797 14326 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4e99abddc89a47eb9bc643bf1f7e2522: Generated new TSK 0
I20260810 04:09:06.829033 14326 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 04:09:07.643191 14283 catalog_manager.cc:1259] Loaded cluster ID: 7b07db4c07f04675837200d59fa7f42a
I20260810 04:09:07.643280 14283 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 42c1e9b618f746d1baafe3a987c3bd31: loading cluster ID for follower catalog manager: success
I20260810 04:09:07.644083 14283 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 42c1e9b618f746d1baafe3a987c3bd31: acquiring CA information for follower catalog manager: success
I20260810 04:09:07.644431 14283 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 42c1e9b618f746d1baafe3a987c3bd31: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 04:09:07.668865 14306 catalog_manager.cc:1259] Loaded cluster ID: 7b07db4c07f04675837200d59fa7f42a
I20260810 04:09:07.668949 14306 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 44c973ce2d7748e0b843e447666bc8ca: loading cluster ID for follower catalog manager: success
I20260810 04:09:07.669759 14306 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 44c973ce2d7748e0b843e447666bc8ca: acquiring CA information for follower catalog manager: success
I20260810 04:09:07.670076 14306 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 44c973ce2d7748e0b843e447666bc8ca: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 04:09:07.877637 31872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskws89Io/build/release/bin/kudu with pid 14020
I20260810 04:09:07.885478 31872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskws89Io/build/release/bin/kudu with pid 14103
I20260810 04:09:07.892097 31872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskws89Io/build/release/bin/kudu with pid 14190
2026-08-10T04:09:07Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (2924 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786334794676589-31872-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 04:09:07 dist-test-slave-8w9v krb5kdc[14343](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 04:09:07 dist-test-slave-8w9v krb5kdc[14343](info): set up 2 sockets
Aug 10 04:09:07 dist-test-slave-8w9v krb5kdc[14343](info): commencing operation
krb5kdc: starting...
W20260810 04:09:09.053094 31872 mini_kdc.cc:121] Time spent starting KDC: real 1.139s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786334794676589-31872-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786334794676589-31872-0/krb5kdc/test-user.keytab.
Aug 10 04:09:09 dist-test-slave-8w9v krb5kdc[14343](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334949, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T04:09:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T04:09:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.32.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.32.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.32.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.62.keytab.
Entry for principal kudu/127.31.32.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.62.keytab.
I20260810 04:09:09.169155 31872 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskws89Io/build/release/bin/kudu
/tmp/dist-test-taskws89Io/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786334794676589-31872-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786334794676589-31872-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786334794676589-31872-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786334794676589-31872-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.32.62:46473
--webserver_interface=127.31.32.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.62.keytab
--principal=kudu/127.31.32.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.32.20:34697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.32.62:46473
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786334794676589-31872-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786334794676589-31872-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786334794676589-31872-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:09:09.250314 14357 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:09:09.250527 14357 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:09:09.250552 14357 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:09:09.251631 14357 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:09:09.251670 14357 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:09:09.251688 14357 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:09:09.251703 14357 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:09:09.251715 14357 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:09:09.253159 14357 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.32.20:34697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786334794676589-31872-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786334794676589-31872-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.32.62:46473
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.62.keytab
--principal=kudu/127.31.32.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.32.62:46473
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786334794676589-31872-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.32.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14357
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786334794676589-31872-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:09:09.253364 14357 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:09:09.253542 14357 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:09:09.258423 14365 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:09:09.258430 14362 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:09:09.258492 14363 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:09:09.258651 14357 server_base.cc:1034] running on GCE node
I20260810 04:09:09.258925 14357 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:09:09.259204 14357 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:09:09.260371 14357 hybrid_clock.cc:648] HybridClock initialized: now 1786334949260362 us; error 34 us; skew 500 ppm
Aug 10 04:09:09 dist-test-slave-8w9v krb5kdc[14343](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334949, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:09:09.262921 14357 init.cc:377] Logged in from keytab as kudu/127.31.32.62@KRBTEST.COM (short username kudu)
I20260810 04:09:09.263401 14357 webserver.cc:460] Webserver started at http://127.31.32.62:45963/ using document root <none> and password file <none>
I20260810 04:09:09.263638 14357 fs_manager.cc:359] Metadata directory not provided
I20260810 04:09:09.263687 14357 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:09:09.263872 14357 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:09:09.265002 14357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786334794676589-31872-0/minicluster-data/master-0/data/instance:
uuid: "9772f6badfae48e5a014bfb867090ce4"
format_stamp: "Formatted at 2026-08-10 04:09:09 on dist-test-slave-8w9v"
server_key: "0421fe9905540013a72ecc565c1e5291"
server_key_iv: "96a55156d80c00bcdb58ed7ae6073e6c"
server_key_version: "encryptionkey@0"
I20260810 04:09:09.265386 14357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786334794676589-31872-0/minicluster-data/master-0/wal/instance:
uuid: "9772f6badfae48e5a014bfb867090ce4"
format_stamp: "Formatted at 2026-08-10 04:09:09 on dist-test-slave-8w9v"
server_key: "0421fe9905540013a72ecc565c1e5291"
server_key_iv: "96a55156d80c00bcdb58ed7ae6073e6c"
server_key_version: "encryptionkey@0"
I20260810 04:09:09.266819 14357 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260810 04:09:09.267616 14372 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:09:09.267902 14357 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 04:09:09.267992 14357 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786334794676589-31872-0/minicluster-data/master-0/data,/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786334794676589-31872-0/minicluster-data/master-0/wal
uuid: "9772f6badfae48e5a014bfb867090ce4"
format_stamp: "Formatted at 2026-08-10 04:09:09 on dist-test-slave-8w9v"
server_key: "0421fe9905540013a72ecc565c1e5291"
server_key_iv: "96a55156d80c00bcdb58ed7ae6073e6c"
server_key_version: "encryptionkey@0"
I20260810 04:09:09.268045 14357 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786334794676589-31872-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786334794676589-31872-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786334794676589-31872-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:09:09.290373 14357 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:09:09.292642 14357 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:09:09.292748 14357 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:09:09.296654 14357 rpc_server.cc:307] RPC server started. Bound to: 127.31.32.62:46473
I20260810 04:09:09.296705 14427 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.32.62:46473 every 8 connection(s)
I20260810 04:09:09.297070 14357 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786334794676589-31872-0/minicluster-data/master-0/data/info.pb
I20260810 04:09:09.297966 14428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:09:09.300206 14428 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9772f6badfae48e5a014bfb867090ce4: Bootstrap starting.
I20260810 04:09:09.300791 14428 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9772f6badfae48e5a014bfb867090ce4: Neither blocks nor log segments found. Creating new log.
I20260810 04:09:09.301087 14428 log.cc:824] T 00000000000000000000000000000000 P 9772f6badfae48e5a014bfb867090ce4: Log is configured to *not* fsync() on all Append() calls
I20260810 04:09:09.301790 14428 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9772f6badfae48e5a014bfb867090ce4: No bootstrap required, opened a new log
I20260810 04:09:09.303277 14428 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9772f6badfae48e5a014bfb867090ce4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9772f6badfae48e5a014bfb867090ce4" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 46473 } }
I20260810 04:09:09.303427 14428 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9772f6badfae48e5a014bfb867090ce4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:09:09.303473 14428 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9772f6badfae48e5a014bfb867090ce4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9772f6badfae48e5a014bfb867090ce4, State: Initialized, Role: FOLLOWER
I20260810 04:09:09.303618 14428 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9772f6badfae48e5a014bfb867090ce4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9772f6badfae48e5a014bfb867090ce4" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 46473 } }
I20260810 04:09:09.303678 31872 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskws89Io/build/release/bin/kudu as pid 14357
I20260810 04:09:09.303710 14428 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9772f6badfae48e5a014bfb867090ce4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 04:09:09.303755 14428 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9772f6badfae48e5a014bfb867090ce4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 04:09:09.303782 31872 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786334794676589-31872-0/minicluster-data/master-0/wal/instance
I20260810 04:09:09.303802 14428 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9772f6badfae48e5a014bfb867090ce4 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:09:09.303970 31872 external_mini_cluster.cc:1428] Setting key 2e0bd4b32f7e2a398d04e67c763478bb
I20260810 04:09:09.304569 14428 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9772f6badfae48e5a014bfb867090ce4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9772f6badfae48e5a014bfb867090ce4" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 46473 } }
I20260810 04:09:09.304682 14428 leader_election.cc:302] T 00000000000000000000000000000000 P 9772f6badfae48e5a014bfb867090ce4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9772f6badfae48e5a014bfb867090ce4; no voters:
I20260810 04:09:09.304845 14428 leader_election.cc:288] T 00000000000000000000000000000000 P 9772f6badfae48e5a014bfb867090ce4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 04:09:09.304890 14434 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9772f6badfae48e5a014bfb867090ce4 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:09:09.305054 14434 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9772f6badfae48e5a014bfb867090ce4 [term 1 LEADER]: Becoming Leader. State: Replica: 9772f6badfae48e5a014bfb867090ce4, State: Running, Role: LEADER
I20260810 04:09:09.305097 14428 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9772f6badfae48e5a014bfb867090ce4 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:09:09.305212 14434 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9772f6badfae48e5a014bfb867090ce4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9772f6badfae48e5a014bfb867090ce4" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 46473 } }
I20260810 04:09:09.305804 14435 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9772f6badfae48e5a014bfb867090ce4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9772f6badfae48e5a014bfb867090ce4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9772f6badfae48e5a014bfb867090ce4" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 46473 } } }
I20260810 04:09:09.305927 14435 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9772f6badfae48e5a014bfb867090ce4 [sys.catalog]: This master's current role is: LEADER
I20260810 04:09:09.305948 14436 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9772f6badfae48e5a014bfb867090ce4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9772f6badfae48e5a014bfb867090ce4. Latest consensus state: current_term: 1 leader_uuid: "9772f6badfae48e5a014bfb867090ce4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9772f6badfae48e5a014bfb867090ce4" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 46473 } } }
I20260810 04:09:09.306033 14436 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9772f6badfae48e5a014bfb867090ce4 [sys.catalog]: This master's current role is: LEADER
I20260810 04:09:09.306299 14443 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 04:09:09.306841 14443 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 04:09:09.308531 14443 catalog_manager.cc:1347] Generated new cluster ID: 027564ee563841a8941d3ecbb299bea8
I20260810 04:09:09.308585 14443 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 10 04:09:09 dist-test-slave-8w9v krb5kdc[14343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334949, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.32.62@KRBTEST.COM
I20260810 04:09:09.315073 14433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:09.304444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35904 (local address 127.31.32.62:46473)
0810 04:09:09.304650 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0810 04:09:09.304659 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 04:09:09.304693 (+ 34us) server_negotiation.cc:407] Connection header received
0810 04:09:09.304808 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:09:09.304815 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:09:09.304871 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:09:09.305015 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:09:09.306037 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:09:09.307614 (+ 1577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:09:09.308539 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:09:09.308996 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:09:09.309110 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:09:09.311279 (+ 2169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:09:09.311282 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:09:09.311283 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:09:09.311294 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:09:09.313432 (+ 2138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:09:09.313867 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:09:09.313874 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:09:09.313874 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:09:09.313969 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:09:09.314173 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:09:09.314174 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:09:09.314175 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:09:09.314404 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:09:09.314569 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0810 04:09:09.314673 (+ 104us) server_negotiation.cc:299] Negotiation successful
0810 04:09:09.314901 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":119,"threads_started":1}
I20260810 04:09:09.316051 31872 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 04:09:09.323195 14443 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 04:09:09.323865 14443 catalog_manager.cc:1504] Loading token signing keys...
I20260810 04:09:09.327704 14443 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9772f6badfae48e5a014bfb867090ce4: Generated new TSK 0
I20260810 04:09:09.327862 14443 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 04:09:09.369992 31872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskws89Io/build/release/bin/kudu with pid 14357
2026-08-10T04:09:09Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1477 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 04:09:09 dist-test-slave-8w9v krb5kdc[14461](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 04:09:09 dist-test-slave-8w9v krb5kdc[14461](info): set up 2 sockets
Aug 10 04:09:09 dist-test-slave-8w9v krb5kdc[14461](info): commencing operation
krb5kdc: starting...
W20260810 04:09:10.457015 31872 mini_kdc.cc:121] Time spent starting KDC: real 1.066s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/krb5kdc/test-user.keytab.
Aug 10 04:09:10 dist-test-slave-8w9v krb5kdc[14461](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334950, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T04:09:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T04:09:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.32.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.32.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.32.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.62.keytab.
Entry for principal kudu/127.31.32.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.62.keytab.
I20260810 04:09:10.578349 31872 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskws89Io/build/release/bin/kudu
/tmp/dist-test-taskws89Io/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.32.62:44907
--webserver_interface=127.31.32.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.62.keytab
--principal=kudu/127.31.32.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.32.20:34365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.32.62:44907
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:09:10.661276 14474 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:09:10.661459 14474 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:09:10.661499 14474 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:09:10.662618 14474 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:09:10.662662 14474 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:09:10.662675 14474 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:09:10.662724 14474 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:09:10.662755 14474 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:09:10.664268 14474 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.32.20:34365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.32.62:44907
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.62.keytab
--principal=kudu/127.31.32.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.32.62:44907
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.32.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14474
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:09:10.664501 14474 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:09:10.664716 14474 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:09:10.669401 14481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:09:10.669459 14483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:09:10.669458 14480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:09:10.669495 14474 server_base.cc:1034] running on GCE node
I20260810 04:09:10.669881 14474 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:09:10.670259 14474 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:09:10.671444 14474 hybrid_clock.cc:648] HybridClock initialized: now 1786334950671433 us; error 30 us; skew 500 ppm
Aug 10 04:09:10 dist-test-slave-8w9v krb5kdc[14461](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334950, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:09:10.673924 14474 init.cc:377] Logged in from keytab as kudu/127.31.32.62@KRBTEST.COM (short username kudu)
I20260810 04:09:10.674307 14474 webserver.cc:460] Webserver started at http://127.31.32.62:44181/ using document root <none> and password file <none>
I20260810 04:09:10.674504 14474 fs_manager.cc:359] Metadata directory not provided
I20260810 04:09:10.674543 14474 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:09:10.674635 14474 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:09:10.675601 14474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/master-0/data/instance:
uuid: "affc5cedaef749e2b7f7f15f09b0ecd0"
format_stamp: "Formatted at 2026-08-10 04:09:10 on dist-test-slave-8w9v"
server_key: "c5e671ecb9c5e9286496d0ca6022cf05"
server_key_iv: "e2e14dd5d26b733c9145fbb549cb1f30"
server_key_version: "encryptionkey@0"
I20260810 04:09:10.675871 14474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/master-0/wal/instance:
uuid: "affc5cedaef749e2b7f7f15f09b0ecd0"
format_stamp: "Formatted at 2026-08-10 04:09:10 on dist-test-slave-8w9v"
server_key: "c5e671ecb9c5e9286496d0ca6022cf05"
server_key_iv: "e2e14dd5d26b733c9145fbb549cb1f30"
server_key_version: "encryptionkey@0"
I20260810 04:09:10.677181 14474 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 04:09:10.677881 14490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:09:10.678031 14474 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 04:09:10.678131 14474 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/master-0/data,/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/master-0/wal
uuid: "affc5cedaef749e2b7f7f15f09b0ecd0"
format_stamp: "Formatted at 2026-08-10 04:09:10 on dist-test-slave-8w9v"
server_key: "c5e671ecb9c5e9286496d0ca6022cf05"
server_key_iv: "e2e14dd5d26b733c9145fbb549cb1f30"
server_key_version: "encryptionkey@0"
I20260810 04:09:10.678200 14474 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:09:10.689610 14474 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:09:10.691780 14474 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:09:10.691941 14474 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:09:10.695906 14474 rpc_server.cc:307] RPC server started. Bound to: 127.31.32.62:44907
I20260810 04:09:10.695950 14553 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.32.62:44907 every 8 connection(s)
I20260810 04:09:10.696362 14474 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/master-0/data/info.pb
I20260810 04:09:10.697300 14554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:09:10.699545 14554 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P affc5cedaef749e2b7f7f15f09b0ecd0: Bootstrap starting.
I20260810 04:09:10.700194 14554 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P affc5cedaef749e2b7f7f15f09b0ecd0: Neither blocks nor log segments found. Creating new log.
I20260810 04:09:10.700477 14554 log.cc:824] T 00000000000000000000000000000000 P affc5cedaef749e2b7f7f15f09b0ecd0: Log is configured to *not* fsync() on all Append() calls
I20260810 04:09:10.701188 14554 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P affc5cedaef749e2b7f7f15f09b0ecd0: No bootstrap required, opened a new log
I20260810 04:09:10.702736 14554 raft_consensus.cc:348] T 00000000000000000000000000000000 P affc5cedaef749e2b7f7f15f09b0ecd0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "affc5cedaef749e2b7f7f15f09b0ecd0" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 44907 } }
I20260810 04:09:10.702873 14554 raft_consensus.cc:374] T 00000000000000000000000000000000 P affc5cedaef749e2b7f7f15f09b0ecd0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:09:10.702898 14554 raft_consensus.cc:729] T 00000000000000000000000000000000 P affc5cedaef749e2b7f7f15f09b0ecd0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affc5cedaef749e2b7f7f15f09b0ecd0, State: Initialized, Role: FOLLOWER
I20260810 04:09:10.703073 14554 consensus_queue.cc:260] T 00000000000000000000000000000000 P affc5cedaef749e2b7f7f15f09b0ecd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "affc5cedaef749e2b7f7f15f09b0ecd0" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 44907 } }
I20260810 04:09:10.703152 14554 raft_consensus.cc:388] T 00000000000000000000000000000000 P affc5cedaef749e2b7f7f15f09b0ecd0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 04:09:10.703200 14554 raft_consensus.cc:482] T 00000000000000000000000000000000 P affc5cedaef749e2b7f7f15f09b0ecd0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 04:09:10.703253 14554 raft_consensus.cc:3037] T 00000000000000000000000000000000 P affc5cedaef749e2b7f7f15f09b0ecd0 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:09:10.703598 31872 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskws89Io/build/release/bin/kudu as pid 14474
I20260810 04:09:10.703727 31872 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/master-0/wal/instance
I20260810 04:09:10.703941 31872 external_mini_cluster.cc:1428] Setting key efcc5bc693efc3024ebcfae04a08e52f
I20260810 04:09:10.704228 14554 raft_consensus.cc:504] T 00000000000000000000000000000000 P affc5cedaef749e2b7f7f15f09b0ecd0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "affc5cedaef749e2b7f7f15f09b0ecd0" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 44907 } }
I20260810 04:09:10.704352 14554 leader_election.cc:302] T 00000000000000000000000000000000 P affc5cedaef749e2b7f7f15f09b0ecd0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: affc5cedaef749e2b7f7f15f09b0ecd0; no voters:
I20260810 04:09:10.704527 14554 leader_election.cc:288] T 00000000000000000000000000000000 P affc5cedaef749e2b7f7f15f09b0ecd0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 04:09:10.704573 14561 raft_consensus.cc:2781] T 00000000000000000000000000000000 P affc5cedaef749e2b7f7f15f09b0ecd0 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:09:10.704731 14561 raft_consensus.cc:686] T 00000000000000000000000000000000 P affc5cedaef749e2b7f7f15f09b0ecd0 [term 1 LEADER]: Becoming Leader. State: Replica: affc5cedaef749e2b7f7f15f09b0ecd0, State: Running, Role: LEADER
I20260810 04:09:10.704772 14554 sys_catalog.cc:564] T 00000000000000000000000000000000 P affc5cedaef749e2b7f7f15f09b0ecd0 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:09:10.704897 14561 consensus_queue.cc:237] T 00000000000000000000000000000000 P affc5cedaef749e2b7f7f15f09b0ecd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "affc5cedaef749e2b7f7f15f09b0ecd0" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 44907 } }
I20260810 04:09:10.705348 14562 sys_catalog.cc:455] T 00000000000000000000000000000000 P affc5cedaef749e2b7f7f15f09b0ecd0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "affc5cedaef749e2b7f7f15f09b0ecd0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "affc5cedaef749e2b7f7f15f09b0ecd0" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 44907 } } }
I20260810 04:09:10.705585 14562 sys_catalog.cc:458] T 00000000000000000000000000000000 P affc5cedaef749e2b7f7f15f09b0ecd0 [sys.catalog]: This master's current role is: LEADER
I20260810 04:09:10.705497 14563 sys_catalog.cc:455] T 00000000000000000000000000000000 P affc5cedaef749e2b7f7f15f09b0ecd0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader affc5cedaef749e2b7f7f15f09b0ecd0. Latest consensus state: current_term: 1 leader_uuid: "affc5cedaef749e2b7f7f15f09b0ecd0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "affc5cedaef749e2b7f7f15f09b0ecd0" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 44907 } } }
I20260810 04:09:10.706055 14563 sys_catalog.cc:458] T 00000000000000000000000000000000 P affc5cedaef749e2b7f7f15f09b0ecd0 [sys.catalog]: This master's current role is: LEADER
I20260810 04:09:10.706213 14571 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 04:09:10.706712 14571 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 04:09:10.708325 14571 catalog_manager.cc:1347] Generated new cluster ID: 3cb9a05ff41b4b4f9928b6cfef7d4d8e
I20260810 04:09:10.708380 14571 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 10 04:09:10 dist-test-slave-8w9v krb5kdc[14461](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334950, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.32.62@KRBTEST.COM
I20260810 04:09:10.716225 14560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:10.704407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50838 (local address 127.31.32.62:44907)
0810 04:09:10.704730 (+ 323us) server_negotiation.cc:206] Beginning negotiation
0810 04:09:10.704739 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 04:09:10.704773 (+ 34us) server_negotiation.cc:407] Connection header received
0810 04:09:10.704912 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:09:10.704921 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:09:10.704977 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:09:10.705127 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:09:10.706038 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:09:10.707166 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:09:10.708110 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:09:10.708681 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:09:10.708780 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:09:10.711554 (+ 2774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:09:10.711558 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:09:10.711559 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:09:10.711571 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:09:10.714431 (+ 2860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:09:10.714849 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:09:10.714852 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:09:10.714852 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:09:10.714971 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:09:10.715171 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:09:10.715172 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:09:10.715173 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:09:10.715475 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:09:10.715639 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0810 04:09:10.715715 (+ 76us) server_negotiation.cc:299] Negotiation successful
0810 04:09:10.715970 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":91,"threads_started":1}
I20260810 04:09:10.723146 14571 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 04:09:10.723656 14571 catalog_manager.cc:1504] Loading token signing keys...
I20260810 04:09:10.727744 14571 catalog_manager.cc:5868] T 00000000000000000000000000000000 P affc5cedaef749e2b7f7f15f09b0ecd0: Generated new TSK 0
I20260810 04:09:10.727902 14571 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.32.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.32.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.32.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.1.keytab.
Entry for principal kudu/127.31.32.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.1.keytab.
I20260810 04:09:10.759240 31872 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskws89Io/build/release/bin/kudu
/tmp/dist-test-taskws89Io/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.32.1:0
--local_ip_for_outbound_sockets=127.31.32.1
--webserver_interface=127.31.32.1
--webserver_port=0
--tserver_master_addrs=127.31.32.62:44907
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.1.keytab
--principal=kudu/127.31.32.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.32.20:34365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:09:10.839883 14583 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:09:10.840075 14583 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:09:10.840096 14583 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:09:10.841176 14583 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:09:10.841215 14583 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:09:10.841240 14583 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.32.1
I20260810 04:09:10.842600 14583 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.32.20:34365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.1.keytab
--principal=kudu/127.31.32.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.32.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.32.1
--webserver_port=0
--tserver_master_addrs=127.31.32.62:44907
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14583
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.32.1
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:09:10.842793 14583 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:09:10.842970 14583 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:09:10.847707 14589 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:09:10.847713 14592 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:09:10.847721 14588 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:09:10.848085 14583 server_base.cc:1034] running on GCE node
I20260810 04:09:10.848315 14583 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:09:10.848522 14583 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:09:10.849682 14583 hybrid_clock.cc:648] HybridClock initialized: now 1786334950849675 us; error 27 us; skew 500 ppm
Aug 10 04:09:10 dist-test-slave-8w9v krb5kdc[14461](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334950, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:09:10.852689 14583 init.cc:377] Logged in from keytab as kudu/127.31.32.1@KRBTEST.COM (short username kudu)
I20260810 04:09:10.853066 14583 webserver.cc:460] Webserver started at http://127.31.32.1:45699/ using document root <none> and password file <none>
I20260810 04:09:10.853238 14583 fs_manager.cc:359] Metadata directory not provided
I20260810 04:09:10.853277 14583 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:09:10.853370 14583 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:09:10.854426 14583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/ts-0/data/instance:
uuid: "df3980e9202d4bb586c7c6b817158342"
format_stamp: "Formatted at 2026-08-10 04:09:10 on dist-test-slave-8w9v"
server_key: "b92f55ed48a4450688ecb93317a5422c"
server_key_iv: "f96d477058c33bdca2dbf94b4555b30f"
server_key_version: "encryptionkey@0"
I20260810 04:09:10.854722 14583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/ts-0/wal/instance:
uuid: "df3980e9202d4bb586c7c6b817158342"
format_stamp: "Formatted at 2026-08-10 04:09:10 on dist-test-slave-8w9v"
server_key: "b92f55ed48a4450688ecb93317a5422c"
server_key_iv: "f96d477058c33bdca2dbf94b4555b30f"
server_key_version: "encryptionkey@0"
I20260810 04:09:10.855933 14583 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 04:09:10.856650 14599 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:09:10.856804 14583 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260810 04:09:10.856904 14583 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/ts-0/data,/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/ts-0/wal
uuid: "df3980e9202d4bb586c7c6b817158342"
format_stamp: "Formatted at 2026-08-10 04:09:10 on dist-test-slave-8w9v"
server_key: "b92f55ed48a4450688ecb93317a5422c"
server_key_iv: "f96d477058c33bdca2dbf94b4555b30f"
server_key_version: "encryptionkey@0"
I20260810 04:09:10.856972 14583 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:09:10.884447 14583 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:09:10.886885 14583 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:09:10.887051 14583 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:09:10.887297 14583 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:09:10.887669 14583 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:09:10.887732 14583 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:09:10.887771 14583 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:09:10.887807 14583 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:09:10.893757 14583 rpc_server.cc:307] RPC server started. Bound to: 127.31.32.1:39207
I20260810 04:09:10.893802 14721 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.32.1:39207 every 8 connection(s)
I20260810 04:09:10.894162 14583 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/ts-0/data/info.pb
Aug 10 04:09:10 dist-test-slave-8w9v krb5kdc[14461](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334950, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.1@KRBTEST.COM for kudu/127.31.32.62@KRBTEST.COM
I20260810 04:09:10.903033 14560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:10.894844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.1:43843 (local address 127.31.32.62:44907)
0810 04:09:10.894923 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0810 04:09:10.894925 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:09:10.895161 (+ 236us) server_negotiation.cc:407] Connection header received
0810 04:09:10.895632 (+ 471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:09:10.895634 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:09:10.895644 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:09:10.895699 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:09:10.896731 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:09:10.897250 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:09:10.898192 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:09:10.898397 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:09:10.898452 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:09:10.900500 (+ 2048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:09:10.900501 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:09:10.900502 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:09:10.900509 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:09:10.901940 (+ 1431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:09:10.902269 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:09:10.902270 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:09:10.902270 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:09:10.902325 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:09:10.902522 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:09:10.902523 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:09:10.902523 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:09:10.902619 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:09:10.902667 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0810 04:09:10.902898 (+ 231us) server_negotiation.cc:299] Negotiation successful
0810 04:09:10.902967 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 04:09:10.903192 14724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:10.894838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:44907 (local address 127.31.32.1:43843)
0810 04:09:10.895019 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0810 04:09:10.895073 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0810 04:09:10.895536 (+ 463us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:09:10.895772 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:09:10.895776 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:09:10.896449 (+ 673us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:09:10.896641 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:09:10.896647 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:09:10.897330 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:09:10.897331 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:09:10.898113 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:09:10.898115 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:09:10.898467 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:09:10.898468 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:09:10.898560 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:09:10.898972 (+ 412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:09:10.898980 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:09:10.900381 (+ 1401us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:09:10.902006 (+ 1625us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:09:10.902011 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:09:10.902013 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:09:10.902209 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:09:10.902379 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:09:10.902379 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:09:10.902379 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:09:10.902471 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:09:10.902668 (+ 197us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:09:10.902672 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:09:10.902777 (+ 105us) client_negotiation.cc:769] Sending connection context
0810 04:09:10.902867 (+ 90us) client_negotiation.cc:240] Negotiation successful
0810 04:09:10.902979 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":94,"threads_started":1}
I20260810 04:09:10.903698 14722 heartbeater.cc:347] Connected to a master server at 127.31.32.62:44907
I20260810 04:09:10.903805 14722 heartbeater.cc:464] Registering TS with master...
I20260810 04:09:10.903983 14722 heartbeater.cc:511] Master 127.31.32.62:44907 requested a full tablet report, sending...
I20260810 04:09:10.904299 31872 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskws89Io/build/release/bin/kudu as pid 14583
I20260810 04:09:10.904436 31872 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/ts-0/wal/instance
I20260810 04:09:10.904631 14509 ts_manager.cc:194] Registered new tserver with Master: df3980e9202d4bb586c7c6b817158342 (127.31.32.1:39207)
I20260810 04:09:10.904660 31872 external_mini_cluster.cc:1428] Setting key 93057fc7628e6f2ca2c693193d8f6806
I20260810 04:09:10.905516 14509 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.32.1@KRBTEST.COM'} at 127.31.32.1:43843
WARNING: no policy specified for kudu/127.31.32.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.32.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.32.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.2.keytab.
Entry for principal kudu/127.31.32.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.2.keytab.
I20260810 04:09:10.929975 31872 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskws89Io/build/release/bin/kudu
/tmp/dist-test-taskws89Io/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.32.2:0
--local_ip_for_outbound_sockets=127.31.32.2
--webserver_interface=127.31.32.2
--webserver_port=0
--tserver_master_addrs=127.31.32.62:44907
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.2.keytab
--principal=kudu/127.31.32.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.32.20:34365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:09:11.009675 14727 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:09:11.009894 14727 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:09:11.009932 14727 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:09:11.011157 14727 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:09:11.011227 14727 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:09:11.011268 14727 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.32.2
I20260810 04:09:11.012742 14727 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.32.20:34365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.2.keytab
--principal=kudu/127.31.32.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.32.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.32.2
--webserver_port=0
--tserver_master_addrs=127.31.32.62:44907
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14727
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.32.2
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:09:11.012996 14727 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:09:11.013207 14727 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:09:11.018108 14733 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:09:11.018100 14736 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:09:11.018101 14734 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:09:11.018316 14727 server_base.cc:1034] running on GCE node
I20260810 04:09:11.018551 14727 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:09:11.018738 14727 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:09:11.019865 14727 hybrid_clock.cc:648] HybridClock initialized: now 1786334951019850 us; error 31 us; skew 500 ppm
Aug 10 04:09:11 dist-test-slave-8w9v krb5kdc[14461](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334951, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:09:11.022239 14727 init.cc:377] Logged in from keytab as kudu/127.31.32.2@KRBTEST.COM (short username kudu)
I20260810 04:09:11.022639 14727 webserver.cc:460] Webserver started at http://127.31.32.2:42345/ using document root <none> and password file <none>
I20260810 04:09:11.022855 14727 fs_manager.cc:359] Metadata directory not provided
I20260810 04:09:11.022918 14727 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:09:11.023053 14727 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:09:11.023998 14727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/ts-1/data/instance:
uuid: "cf99f2b9d85949f8ae4e5bbe8021050b"
format_stamp: "Formatted at 2026-08-10 04:09:11 on dist-test-slave-8w9v"
server_key: "80e271ba77cc9983241470c2b1abf7a9"
server_key_iv: "945fbf70d98a4ebfbc2c8275c88b2169"
server_key_version: "encryptionkey@0"
I20260810 04:09:11.024379 14727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/ts-1/wal/instance:
uuid: "cf99f2b9d85949f8ae4e5bbe8021050b"
format_stamp: "Formatted at 2026-08-10 04:09:11 on dist-test-slave-8w9v"
server_key: "80e271ba77cc9983241470c2b1abf7a9"
server_key_iv: "945fbf70d98a4ebfbc2c8275c88b2169"
server_key_version: "encryptionkey@0"
I20260810 04:09:11.025650 14727 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260810 04:09:11.026331 14745 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:09:11.026487 14727 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 04:09:11.026582 14727 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/ts-1/data,/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/ts-1/wal
uuid: "cf99f2b9d85949f8ae4e5bbe8021050b"
format_stamp: "Formatted at 2026-08-10 04:09:11 on dist-test-slave-8w9v"
server_key: "80e271ba77cc9983241470c2b1abf7a9"
server_key_iv: "945fbf70d98a4ebfbc2c8275c88b2169"
server_key_version: "encryptionkey@0"
I20260810 04:09:11.026649 14727 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:09:11.037345 14727 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:09:11.038960 14727 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:09:11.039108 14727 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:09:11.039305 14727 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:09:11.039597 14727 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:09:11.039646 14727 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:09:11.039685 14727 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:09:11.039711 14727 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:09:11.045962 14727 rpc_server.cc:307] RPC server started. Bound to: 127.31.32.2:34627
I20260810 04:09:11.046066 14867 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.32.2:34627 every 8 connection(s)
I20260810 04:09:11.046365 14727 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/ts-1/data/info.pb
Aug 10 04:09:11 dist-test-slave-8w9v krb5kdc[14461](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334951, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.2@KRBTEST.COM for kudu/127.31.32.62@KRBTEST.COM
I20260810 04:09:11.054960 31872 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskws89Io/build/release/bin/kudu as pid 14727
I20260810 04:09:11.055099 31872 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/ts-1/wal/instance
I20260810 04:09:11.055337 31872 external_mini_cluster.cc:1428] Setting key aac85b905de6b3a90e3e5ae89b81dd83
I20260810 04:09:11.055691 14560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:11.048091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.2:54975 (local address 127.31.32.62:44907)
0810 04:09:11.048179 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0810 04:09:11.048181 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:09:11.048194 (+ 13us) server_negotiation.cc:407] Connection header received
0810 04:09:11.048651 (+ 457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:09:11.048652 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:09:11.048662 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:09:11.048718 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:09:11.049267 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:09:11.049777 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:09:11.050727 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:09:11.050911 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:09:11.050962 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:09:11.053069 (+ 2107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:09:11.053071 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:09:11.053071 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:09:11.053078 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:09:11.054318 (+ 1240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:09:11.054669 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:09:11.054670 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:09:11.054671 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:09:11.054719 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:09:11.055002 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:09:11.055003 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:09:11.055003 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:09:11.055151 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:09:11.055217 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0810 04:09:11.055505 (+ 288us) server_negotiation.cc:299] Negotiation successful
0810 04:09:11.055598 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260810 04:09:11.055886 14871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:11.047087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:44907 (local address 127.31.32.2:54975)
0810 04:09:11.047237 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0810 04:09:11.047277 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0810 04:09:11.048556 (+ 1279us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:09:11.048772 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:09:11.048775 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:09:11.049016 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:09:11.049198 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:09:11.049202 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:09:11.049850 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:09:11.049851 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:09:11.050653 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:09:11.050655 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:09:11.050978 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:09:11.050979 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:09:11.051079 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:09:11.051623 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:09:11.051632 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:09:11.052976 (+ 1344us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:09:11.054400 (+ 1424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:09:11.054406 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:09:11.054407 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:09:11.054608 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:09:11.054785 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:09:11.054786 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:09:11.054786 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:09:11.054926 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:09:11.055210 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:09:11.055217 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:09:11.055366 (+ 149us) client_negotiation.cc:769] Sending connection context
0810 04:09:11.055483 (+ 117us) client_negotiation.cc:240] Negotiation successful
0810 04:09:11.055638 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":85,"threads_started":1}
I20260810 04:09:11.056407 14869 heartbeater.cc:347] Connected to a master server at 127.31.32.62:44907
I20260810 04:09:11.056500 14869 heartbeater.cc:464] Registering TS with master...
I20260810 04:09:11.056691 14869 heartbeater.cc:511] Master 127.31.32.62:44907 requested a full tablet report, sending...
I20260810 04:09:11.057040 14509 ts_manager.cc:194] Registered new tserver with Master: cf99f2b9d85949f8ae4e5bbe8021050b (127.31.32.2:34627)
I20260810 04:09:11.057415 14509 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.32.2@KRBTEST.COM'} at 127.31.32.2:54975
WARNING: no policy specified for kudu/127.31.32.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.32.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.32.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.3.keytab.
Entry for principal kudu/127.31.32.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.3.keytab.
I20260810 04:09:11.080760 31872 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskws89Io/build/release/bin/kudu
/tmp/dist-test-taskws89Io/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.32.3:0
--local_ip_for_outbound_sockets=127.31.32.3
--webserver_interface=127.31.32.3
--webserver_port=0
--tserver_master_addrs=127.31.32.62:44907
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.3.keytab
--principal=kudu/127.31.32.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.32.20:34365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:09:11.162032 14874 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:09:11.162240 14874 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:09:11.162262 14874 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:09:11.163300 14874 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:09:11.163338 14874 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:09:11.163363 14874 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.32.3
I20260810 04:09:11.164745 14874 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.32.20:34365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.3.keytab
--principal=kudu/127.31.32.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.32.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.32.3
--webserver_port=0
--tserver_master_addrs=127.31.32.62:44907
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14874
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.32.3
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:09:11.164935 14874 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:09:11.165099 14874 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:09:11.170250 14880 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:09:11.170244 14882 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:09:11.170222 14879 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:09:11.170265 14874 server_base.cc:1034] running on GCE node
I20260810 04:09:11.170662 14874 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:09:11.170907 14874 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:09:11.172051 14874 hybrid_clock.cc:648] HybridClock initialized: now 1786334951172032 us; error 31 us; skew 500 ppm
Aug 10 04:09:11 dist-test-slave-8w9v krb5kdc[14461](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334951, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:09:11.174530 14874 init.cc:377] Logged in from keytab as kudu/127.31.32.3@KRBTEST.COM (short username kudu)
I20260810 04:09:11.174944 14874 webserver.cc:460] Webserver started at http://127.31.32.3:33751/ using document root <none> and password file <none>
I20260810 04:09:11.175163 14874 fs_manager.cc:359] Metadata directory not provided
I20260810 04:09:11.175228 14874 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:09:11.175375 14874 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:09:11.176499 14874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/ts-2/data/instance:
uuid: "2195f4b573b84fda85724f693eda2a36"
format_stamp: "Formatted at 2026-08-10 04:09:11 on dist-test-slave-8w9v"
server_key: "70cd799f0634144a7b3b59ed05934c23"
server_key_iv: "3673ef5a08d4dc00e0b0e6746192deda"
server_key_version: "encryptionkey@0"
I20260810 04:09:11.176844 14874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/ts-2/wal/instance:
uuid: "2195f4b573b84fda85724f693eda2a36"
format_stamp: "Formatted at 2026-08-10 04:09:11 on dist-test-slave-8w9v"
server_key: "70cd799f0634144a7b3b59ed05934c23"
server_key_iv: "3673ef5a08d4dc00e0b0e6746192deda"
server_key_version: "encryptionkey@0"
I20260810 04:09:11.178115 14874 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 04:09:11.178798 14890 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:09:11.178957 14874 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 04:09:11.179054 14874 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/ts-2/data,/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/ts-2/wal
uuid: "2195f4b573b84fda85724f693eda2a36"
format_stamp: "Formatted at 2026-08-10 04:09:11 on dist-test-slave-8w9v"
server_key: "70cd799f0634144a7b3b59ed05934c23"
server_key_iv: "3673ef5a08d4dc00e0b0e6746192deda"
server_key_version: "encryptionkey@0"
I20260810 04:09:11.179121 14874 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:09:11.191759 14874 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:09:11.193666 14874 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:09:11.193813 14874 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:09:11.194028 14874 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:09:11.194314 14874 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:09:11.194370 14874 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:09:11.194410 14874 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:09:11.194446 14874 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:09:11.200289 14874 rpc_server.cc:307] RPC server started. Bound to: 127.31.32.3:46777
I20260810 04:09:11.200358 15011 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.32.3:46777 every 8 connection(s)
I20260810 04:09:11.200678 14874 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/ts-2/data/info.pb
I20260810 04:09:11.206094 31872 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskws89Io/build/release/bin/kudu as pid 14874
I20260810 04:09:11.206209 31872 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/ts-2/wal/instance
I20260810 04:09:11.206419 31872 external_mini_cluster.cc:1428] Setting key 5ae753b52c1e3e60511173c72fb96609
Aug 10 04:09:11 dist-test-slave-8w9v krb5kdc[14461](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334951, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.3@KRBTEST.COM for kudu/127.31.32.62@KRBTEST.COM
I20260810 04:09:11.210182 14560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:11.201482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.3:49455 (local address 127.31.32.62:44907)
0810 04:09:11.201544 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0810 04:09:11.201546 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:09:11.201758 (+ 212us) server_negotiation.cc:407] Connection header received
0810 04:09:11.202291 (+ 533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:09:11.202293 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:09:11.202306 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:09:11.202385 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:09:11.203327 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:09:11.203825 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:09:11.204916 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:09:11.205186 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:09:11.205264 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:09:11.207641 (+ 2377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:09:11.207644 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:09:11.207644 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:09:11.207652 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:09:11.208847 (+ 1195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:09:11.209224 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:09:11.209225 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:09:11.209225 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:09:11.209286 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:09:11.209561 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:09:11.209562 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:09:11.209562 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:09:11.209679 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:09:11.209745 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0810 04:09:11.210014 (+ 269us) server_negotiation.cc:299] Negotiation successful
0810 04:09:11.210092 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260810 04:09:11.210374 15014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:11.201441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:44907 (local address 127.31.32.3:49455)
0810 04:09:11.201624 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0810 04:09:11.201683 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0810 04:09:11.202179 (+ 496us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:09:11.202461 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:09:11.202465 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:09:11.203035 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:09:11.203218 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:09:11.203223 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:09:11.203916 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:09:11.203917 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:09:11.204806 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:09:11.204810 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:09:11.205277 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:09:11.205278 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:09:11.205400 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:09:11.205801 (+ 401us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:09:11.205809 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:09:11.207553 (+ 1744us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:09:11.208936 (+ 1383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:09:11.208943 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:09:11.208945 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:09:11.209151 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:09:11.209364 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:09:11.209364 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:09:11.209365 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:09:11.209490 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:09:11.209748 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:09:11.209756 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:09:11.209889 (+ 133us) client_negotiation.cc:769] Sending connection context
0810 04:09:11.209990 (+ 101us) client_negotiation.cc:240] Negotiation successful
0810 04:09:11.210139 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":94,"threads_started":1}
I20260810 04:09:11.210836 15012 heartbeater.cc:347] Connected to a master server at 127.31.32.62:44907
I20260810 04:09:11.210945 15012 heartbeater.cc:464] Registering TS with master...
I20260810 04:09:11.211113 15012 heartbeater.cc:511] Master 127.31.32.62:44907 requested a full tablet report, sending...
I20260810 04:09:11.211472 14509 ts_manager.cc:194] Registered new tserver with Master: 2195f4b573b84fda85724f693eda2a36 (127.31.32.3:46777)
I20260810 04:09:11.211827 14509 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.32.3@KRBTEST.COM'} at 127.31.32.3:49455
I20260810 04:09:11.218107 31872 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 04:09:11.225063 14560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:11.219461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50852 (local address 127.31.32.62:44907)
0810 04:09:11.219529 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0810 04:09:11.219531 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:09:11.219604 (+ 73us) server_negotiation.cc:407] Connection header received
0810 04:09:11.219678 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:09:11.219679 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:09:11.219688 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:09:11.219750 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:09:11.220320 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:09:11.220749 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:09:11.221465 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:09:11.221643 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:09:11.221720 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:09:11.222836 (+ 1116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:09:11.222837 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:09:11.222837 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:09:11.222844 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:09:11.223958 (+ 1114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:09:11.224327 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:09:11.224329 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:09:11.224329 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:09:11.224384 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:09:11.224574 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:09:11.224575 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:09:11.224575 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:09:11.224704 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:09:11.224802 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0810 04:09:11.224901 (+ 99us) server_negotiation.cc:299] Negotiation successful
0810 04:09:11.224972 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260810 04:09:11.226387 14509 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50852:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 04:09:11.227963 14509 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 04:09:11.234217 15024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:11.230148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.3:46777 (local address 127.0.0.1:49962)
0810 04:09:11.230323 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0810 04:09:11.230363 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0810 04:09:11.230431 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:09:11.230648 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:09:11.230651 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:09:11.230653 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:09:11.230714 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:09:11.230716 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:09:11.232091 (+ 1375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:09:11.232092 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:09:11.233118 (+ 1026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:09:11.233121 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:09:11.233899 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:09:11.233900 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:09:11.233977 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:09:11.233991 (+ 14us) client_negotiation.cc:769] Sending connection context
0810 04:09:11.234080 (+ 89us) client_negotiation.cc:240] Negotiation successful
0810 04:09:11.234138 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"mutex_wait_us":36,"thread_start_us":56,"threads_started":1}
I20260810 04:09:11.234225 15025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:11.230211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49962 (local address 127.31.32.3:46777)
0810 04:09:11.230420 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0810 04:09:11.230425 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 04:09:11.230436 (+ 11us) server_negotiation.cc:407] Connection header received
0810 04:09:11.230557 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:09:11.230560 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:09:11.230575 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:09:11.230646 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:09:11.230854 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:09:11.231952 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:09:11.233198 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:09:11.233811 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:09:11.233887 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:09:11.233941 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0810 04:09:11.234102 (+ 161us) server_negotiation.cc:299] Negotiation successful
0810 04:09:11.234143 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":93,"threads_started":1}
I20260810 04:09:11.234665 15023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:11.230153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.2:34627 (local address 127.0.0.1:54424)
0810 04:09:11.230475 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0810 04:09:11.230485 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 04:09:11.230526 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:09:11.230759 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:09:11.230760 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:09:11.230762 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:09:11.230803 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:09:11.230804 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:09:11.232075 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:09:11.232076 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:09:11.233393 (+ 1317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:09:11.233396 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:09:11.234214 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:09:11.234215 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:09:11.234278 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:09:11.234281 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 04:09:11.234542 (+ 261us) client_negotiation.cc:240] Negotiation successful
0810 04:09:11.234603 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"mutex_wait_us":96,"thread_start_us":87,"threads_started":1}
I20260810 04:09:11.234712 15026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:11.230258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54424 (local address 127.31.32.2:34627)
0810 04:09:11.230593 (+ 335us) server_negotiation.cc:206] Beginning negotiation
0810 04:09:11.230598 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 04:09:11.230609 (+ 11us) server_negotiation.cc:407] Connection header received
0810 04:09:11.230651 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:09:11.230652 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:09:11.230667 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:09:11.230762 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:09:11.230896 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:09:11.231981 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:09:11.233498 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:09:11.234074 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:09:11.234131 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:09:11.234182 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0810 04:09:11.234579 (+ 397us) server_negotiation.cc:299] Negotiation successful
0810 04:09:11.234632 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":103,"threads_started":1}
I20260810 04:09:11.235004 14940 tablet_service.cc:1467] Processing CreateTablet for tablet ccc0fe3d4a234214beec3dd0e3e95344 (DEFAULT_TABLE table=test-table [id=cfc46bc4c2b54273aa6f612bb39b5cf8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:09:11.235122 15027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:11.230363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.1:39207 (local address 127.0.0.1:44328)
0810 04:09:11.230692 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0810 04:09:11.230703 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 04:09:11.230747 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:09:11.231041 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:09:11.231042 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:09:11.231044 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:09:11.231085 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:09:11.231087 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:09:11.231970 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:09:11.231971 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:09:11.233395 (+ 1424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:09:11.233396 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:09:11.234628 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:09:11.234629 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:09:11.234689 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:09:11.234693 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 04:09:11.234974 (+ 281us) client_negotiation.cc:240] Negotiation successful
0810 04:09:11.235036 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"mutex_wait_us":108,"thread_start_us":69,"threads_started":1}
I20260810 04:09:11.235373 14940 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccc0fe3d4a234214beec3dd0e3e95344. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:09:11.235522 15028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:11.230544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44328 (local address 127.31.32.1:39207)
0810 04:09:11.230875 (+ 331us) server_negotiation.cc:206] Beginning negotiation
0810 04:09:11.230879 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 04:09:11.230889 (+ 10us) server_negotiation.cc:407] Connection header received
0810 04:09:11.230927 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:09:11.230930 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:09:11.230944 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:09:11.231159 (+ 215us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:09:11.231216 (+ 57us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:09:11.231867 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:09:11.233480 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:09:11.234220 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:09:11.235282 (+ 1062us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:09:11.235353 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0810 04:09:11.235379 (+ 26us) server_negotiation.cc:299] Negotiation successful
0810 04:09:11.235438 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":78,"threads_started":1}
I20260810 04:09:11.236218 14652 tablet_service.cc:1467] Processing CreateTablet for tablet ccc0fe3d4a234214beec3dd0e3e95344 (DEFAULT_TABLE table=test-table [id=cfc46bc4c2b54273aa6f612bb39b5cf8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:09:11.236526 14652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccc0fe3d4a234214beec3dd0e3e95344. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:09:11.238029 15029 tablet_bootstrap.cc:492] T ccc0fe3d4a234214beec3dd0e3e95344 P 2195f4b573b84fda85724f693eda2a36: Bootstrap starting.
I20260810 04:09:11.238765 15029 tablet_bootstrap.cc:654] T ccc0fe3d4a234214beec3dd0e3e95344 P 2195f4b573b84fda85724f693eda2a36: Neither blocks nor log segments found. Creating new log.
I20260810 04:09:11.238729 14797 tablet_service.cc:1467] Processing CreateTablet for tablet ccc0fe3d4a234214beec3dd0e3e95344 (DEFAULT_TABLE table=test-table [id=cfc46bc4c2b54273aa6f612bb39b5cf8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:09:11.239063 14797 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccc0fe3d4a234214beec3dd0e3e95344. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:09:11.239336 15029 log.cc:824] T ccc0fe3d4a234214beec3dd0e3e95344 P 2195f4b573b84fda85724f693eda2a36: Log is configured to *not* fsync() on all Append() calls
I20260810 04:09:11.239632 15030 tablet_bootstrap.cc:492] T ccc0fe3d4a234214beec3dd0e3e95344 P df3980e9202d4bb586c7c6b817158342: Bootstrap starting.
I20260810 04:09:11.240092 15029 tablet_bootstrap.cc:492] T ccc0fe3d4a234214beec3dd0e3e95344 P 2195f4b573b84fda85724f693eda2a36: No bootstrap required, opened a new log
I20260810 04:09:11.240190 15029 ts_tablet_manager.cc:1397] T ccc0fe3d4a234214beec3dd0e3e95344 P 2195f4b573b84fda85724f693eda2a36: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 04:09:11.240317 15030 tablet_bootstrap.cc:654] T ccc0fe3d4a234214beec3dd0e3e95344 P df3980e9202d4bb586c7c6b817158342: Neither blocks nor log segments found. Creating new log.
I20260810 04:09:11.240783 15030 log.cc:824] T ccc0fe3d4a234214beec3dd0e3e95344 P df3980e9202d4bb586c7c6b817158342: Log is configured to *not* fsync() on all Append() calls
I20260810 04:09:11.241518 15030 tablet_bootstrap.cc:492] T ccc0fe3d4a234214beec3dd0e3e95344 P df3980e9202d4bb586c7c6b817158342: No bootstrap required, opened a new log
I20260810 04:09:11.241585 15030 ts_tablet_manager.cc:1397] T ccc0fe3d4a234214beec3dd0e3e95344 P df3980e9202d4bb586c7c6b817158342: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 04:09:11.242276 15029 raft_consensus.cc:348] T ccc0fe3d4a234214beec3dd0e3e95344 P 2195f4b573b84fda85724f693eda2a36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2195f4b573b84fda85724f693eda2a36" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 46777 } } peers { permanent_uuid: "cf99f2b9d85949f8ae4e5bbe8021050b" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 34627 } } peers { permanent_uuid: "df3980e9202d4bb586c7c6b817158342" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 39207 } }
I20260810 04:09:11.242455 15029 raft_consensus.cc:374] T ccc0fe3d4a234214beec3dd0e3e95344 P 2195f4b573b84fda85724f693eda2a36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:09:11.242395 15033 tablet_bootstrap.cc:492] T ccc0fe3d4a234214beec3dd0e3e95344 P cf99f2b9d85949f8ae4e5bbe8021050b: Bootstrap starting.
I20260810 04:09:11.242511 15029 raft_consensus.cc:729] T ccc0fe3d4a234214beec3dd0e3e95344 P 2195f4b573b84fda85724f693eda2a36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2195f4b573b84fda85724f693eda2a36, State: Initialized, Role: FOLLOWER
I20260810 04:09:11.242631 15029 consensus_queue.cc:260] T ccc0fe3d4a234214beec3dd0e3e95344 P 2195f4b573b84fda85724f693eda2a36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2195f4b573b84fda85724f693eda2a36" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 46777 } } peers { permanent_uuid: "cf99f2b9d85949f8ae4e5bbe8021050b" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 34627 } } peers { permanent_uuid: "df3980e9202d4bb586c7c6b817158342" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 39207 } }
I20260810 04:09:11.242913 15029 ts_tablet_manager.cc:1428] T ccc0fe3d4a234214beec3dd0e3e95344 P 2195f4b573b84fda85724f693eda2a36: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260810 04:09:11.243040 15012 heartbeater.cc:503] Master 127.31.32.62:44907 was elected leader, sending a full tablet report...
I20260810 04:09:11.243125 15033 tablet_bootstrap.cc:654] T ccc0fe3d4a234214beec3dd0e3e95344 P cf99f2b9d85949f8ae4e5bbe8021050b: Neither blocks nor log segments found. Creating new log.
I20260810 04:09:11.243384 15030 raft_consensus.cc:348] T ccc0fe3d4a234214beec3dd0e3e95344 P df3980e9202d4bb586c7c6b817158342 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2195f4b573b84fda85724f693eda2a36" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 46777 } } peers { permanent_uuid: "cf99f2b9d85949f8ae4e5bbe8021050b" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 34627 } } peers { permanent_uuid: "df3980e9202d4bb586c7c6b817158342" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 39207 } }
I20260810 04:09:11.243590 15030 raft_consensus.cc:374] T ccc0fe3d4a234214beec3dd0e3e95344 P df3980e9202d4bb586c7c6b817158342 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:09:11.243628 15030 raft_consensus.cc:729] T ccc0fe3d4a234214beec3dd0e3e95344 P df3980e9202d4bb586c7c6b817158342 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df3980e9202d4bb586c7c6b817158342, State: Initialized, Role: FOLLOWER
I20260810 04:09:11.243713 15033 log.cc:824] T ccc0fe3d4a234214beec3dd0e3e95344 P cf99f2b9d85949f8ae4e5bbe8021050b: Log is configured to *not* fsync() on all Append() calls
I20260810 04:09:11.243772 15030 consensus_queue.cc:260] T ccc0fe3d4a234214beec3dd0e3e95344 P df3980e9202d4bb586c7c6b817158342 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2195f4b573b84fda85724f693eda2a36" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 46777 } } peers { permanent_uuid: "cf99f2b9d85949f8ae4e5bbe8021050b" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 34627 } } peers { permanent_uuid: "df3980e9202d4bb586c7c6b817158342" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 39207 } }
I20260810 04:09:11.244028 15030 ts_tablet_manager.cc:1428] T ccc0fe3d4a234214beec3dd0e3e95344 P df3980e9202d4bb586c7c6b817158342: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260810 04:09:11.244194 14722 heartbeater.cc:503] Master 127.31.32.62:44907 was elected leader, sending a full tablet report...
I20260810 04:09:11.244877 15033 tablet_bootstrap.cc:492] T ccc0fe3d4a234214beec3dd0e3e95344 P cf99f2b9d85949f8ae4e5bbe8021050b: No bootstrap required, opened a new log
I20260810 04:09:11.244961 15033 ts_tablet_manager.cc:1397] T ccc0fe3d4a234214beec3dd0e3e95344 P cf99f2b9d85949f8ae4e5bbe8021050b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 04:09:11.246495 15033 raft_consensus.cc:348] T ccc0fe3d4a234214beec3dd0e3e95344 P cf99f2b9d85949f8ae4e5bbe8021050b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2195f4b573b84fda85724f693eda2a36" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 46777 } } peers { permanent_uuid: "cf99f2b9d85949f8ae4e5bbe8021050b" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 34627 } } peers { permanent_uuid: "df3980e9202d4bb586c7c6b817158342" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 39207 } }
I20260810 04:09:11.246611 15033 raft_consensus.cc:374] T ccc0fe3d4a234214beec3dd0e3e95344 P cf99f2b9d85949f8ae4e5bbe8021050b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:09:11.246632 15033 raft_consensus.cc:729] T ccc0fe3d4a234214beec3dd0e3e95344 P cf99f2b9d85949f8ae4e5bbe8021050b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf99f2b9d85949f8ae4e5bbe8021050b, State: Initialized, Role: FOLLOWER
I20260810 04:09:11.246753 15033 consensus_queue.cc:260] T ccc0fe3d4a234214beec3dd0e3e95344 P cf99f2b9d85949f8ae4e5bbe8021050b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2195f4b573b84fda85724f693eda2a36" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 46777 } } peers { permanent_uuid: "cf99f2b9d85949f8ae4e5bbe8021050b" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 34627 } } peers { permanent_uuid: "df3980e9202d4bb586c7c6b817158342" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 39207 } }
I20260810 04:09:11.247049 15033 ts_tablet_manager.cc:1428] T ccc0fe3d4a234214beec3dd0e3e95344 P cf99f2b9d85949f8ae4e5bbe8021050b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260810 04:09:11.247051 14869 heartbeater.cc:503] Master 127.31.32.62:44907 was elected leader, sending a full tablet report...
I20260810 04:09:11.259022 15037 raft_consensus.cc:482] T ccc0fe3d4a234214beec3dd0e3e95344 P df3980e9202d4bb586c7c6b817158342 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 04:09:11.259109 15037 raft_consensus.cc:504] T ccc0fe3d4a234214beec3dd0e3e95344 P df3980e9202d4bb586c7c6b817158342 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2195f4b573b84fda85724f693eda2a36" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 46777 } } peers { permanent_uuid: "cf99f2b9d85949f8ae4e5bbe8021050b" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 34627 } } peers { permanent_uuid: "df3980e9202d4bb586c7c6b817158342" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 39207 } }
I20260810 04:09:11.259481 15037 leader_election.cc:288] T ccc0fe3d4a234214beec3dd0e3e95344 P df3980e9202d4bb586c7c6b817158342 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2195f4b573b84fda85724f693eda2a36 (127.31.32.3:46777), cf99f2b9d85949f8ae4e5bbe8021050b (127.31.32.2:34627)
I20260810 04:09:11.262310 15026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:11.259706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.1:38811 (local address 127.31.32.2:34627)
0810 04:09:11.259746 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0810 04:09:11.259748 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:09:11.259758 (+ 10us) server_negotiation.cc:407] Connection header received
0810 04:09:11.259841 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:09:11.259841 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:09:11.259850 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:09:11.259920 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:09:11.260073 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:09:11.260600 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:09:11.261584 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:09:11.261971 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:09:11.262051 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:09:11.262062 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0810 04:09:11.262208 (+ 146us) server_negotiation.cc:299] Negotiation successful
0810 04:09:11.262242 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260810 04:09:11.262324 14724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:11.259647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.2:34627 (local address 127.31.32.1:38811)
0810 04:09:11.259711 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0810 04:09:11.259724 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 04:09:11.259771 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:09:11.259926 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:09:11.259928 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:09:11.259930 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:09:11.260007 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:09:11.260009 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:09:11.260681 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:09:11.260682 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:09:11.261495 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:09:11.261498 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:09:11.262051 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:09:11.262051 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:09:11.262138 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:09:11.262141 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 04:09:11.262203 (+ 62us) client_negotiation.cc:240] Negotiation successful
0810 04:09:11.262243 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44,"mutex_wait_us":28}
I20260810 04:09:11.262373 15025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:11.259701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.1:60963 (local address 127.31.32.3:46777)
0810 04:09:11.259746 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0810 04:09:11.259748 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:09:11.259953 (+ 205us) server_negotiation.cc:407] Connection header received
0810 04:09:11.260027 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:09:11.260028 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:09:11.260039 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:09:11.260103 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:09:11.260267 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:09:11.260718 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:09:11.261671 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:09:11.262027 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:09:11.262101 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:09:11.262111 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0810 04:09:11.262260 (+ 149us) server_negotiation.cc:299] Negotiation successful
0810 04:09:11.262295 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260810 04:09:11.262634 14817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccc0fe3d4a234214beec3dd0e3e95344" candidate_uuid: "df3980e9202d4bb586c7c6b817158342" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf99f2b9d85949f8ae4e5bbe8021050b" is_pre_election: true
I20260810 04:09:11.262774 15039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:11.259651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.3:46777 (local address 127.31.32.1:60963)
0810 04:09:11.259891 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0810 04:09:11.259898 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0810 04:09:11.259956 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:09:11.260107 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:09:11.260108 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:09:11.260110 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:09:11.260194 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:09:11.260196 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:09:11.260816 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:09:11.260818 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:09:11.261583 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:09:11.261585 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:09:11.262115 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:09:11.262116 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:09:11.262191 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:09:11.262194 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 04:09:11.262255 (+ 61us) client_negotiation.cc:240] Negotiation successful
0810 04:09:11.262294 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"mutex_wait_us":53,"thread_start_us":97,"threads_started":1}
I20260810 04:09:11.262843 14817 raft_consensus.cc:2454] T ccc0fe3d4a234214beec3dd0e3e95344 P cf99f2b9d85949f8ae4e5bbe8021050b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3980e9202d4bb586c7c6b817158342 in term 0.
I20260810 04:09:11.263163 14961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccc0fe3d4a234214beec3dd0e3e95344" candidate_uuid: "df3980e9202d4bb586c7c6b817158342" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2195f4b573b84fda85724f693eda2a36" is_pre_election: true
I20260810 04:09:11.263324 14961 raft_consensus.cc:2454] T ccc0fe3d4a234214beec3dd0e3e95344 P 2195f4b573b84fda85724f693eda2a36 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3980e9202d4bb586c7c6b817158342 in term 0.
I20260810 04:09:11.263530 14600 leader_election.cc:302] T ccc0fe3d4a234214beec3dd0e3e95344 P df3980e9202d4bb586c7c6b817158342 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cf99f2b9d85949f8ae4e5bbe8021050b, df3980e9202d4bb586c7c6b817158342; no voters:
I20260810 04:09:11.263689 15037 raft_consensus.cc:2781] T ccc0fe3d4a234214beec3dd0e3e95344 P df3980e9202d4bb586c7c6b817158342 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 04:09:11.263728 15037 raft_consensus.cc:482] T ccc0fe3d4a234214beec3dd0e3e95344 P df3980e9202d4bb586c7c6b817158342 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 04:09:11.263743 15037 raft_consensus.cc:3037] T ccc0fe3d4a234214beec3dd0e3e95344 P df3980e9202d4bb586c7c6b817158342 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:09:11.264473 15037 raft_consensus.cc:504] T ccc0fe3d4a234214beec3dd0e3e95344 P df3980e9202d4bb586c7c6b817158342 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2195f4b573b84fda85724f693eda2a36" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 46777 } } peers { permanent_uuid: "cf99f2b9d85949f8ae4e5bbe8021050b" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 34627 } } peers { permanent_uuid: "df3980e9202d4bb586c7c6b817158342" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 39207 } }
I20260810 04:09:11.264607 15037 leader_election.cc:288] T ccc0fe3d4a234214beec3dd0e3e95344 P df3980e9202d4bb586c7c6b817158342 [CANDIDATE]: Term 1 election: Requested vote from peers 2195f4b573b84fda85724f693eda2a36 (127.31.32.3:46777), cf99f2b9d85949f8ae4e5bbe8021050b (127.31.32.2:34627)
I20260810 04:09:11.264770 14817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccc0fe3d4a234214beec3dd0e3e95344" candidate_uuid: "df3980e9202d4bb586c7c6b817158342" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf99f2b9d85949f8ae4e5bbe8021050b"
I20260810 04:09:11.264770 14961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccc0fe3d4a234214beec3dd0e3e95344" candidate_uuid: "df3980e9202d4bb586c7c6b817158342" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2195f4b573b84fda85724f693eda2a36"
I20260810 04:09:11.264847 14817 raft_consensus.cc:3037] T ccc0fe3d4a234214beec3dd0e3e95344 P cf99f2b9d85949f8ae4e5bbe8021050b [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:09:11.264847 14961 raft_consensus.cc:3037] T ccc0fe3d4a234214beec3dd0e3e95344 P 2195f4b573b84fda85724f693eda2a36 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:09:11.265704 14817 raft_consensus.cc:2454] T ccc0fe3d4a234214beec3dd0e3e95344 P cf99f2b9d85949f8ae4e5bbe8021050b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df3980e9202d4bb586c7c6b817158342 in term 1.
I20260810 04:09:11.265704 14961 raft_consensus.cc:2454] T ccc0fe3d4a234214beec3dd0e3e95344 P 2195f4b573b84fda85724f693eda2a36 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df3980e9202d4bb586c7c6b817158342 in term 1.
I20260810 04:09:11.265862 14600 leader_election.cc:302] T ccc0fe3d4a234214beec3dd0e3e95344 P df3980e9202d4bb586c7c6b817158342 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cf99f2b9d85949f8ae4e5bbe8021050b, df3980e9202d4bb586c7c6b817158342; no voters:
I20260810 04:09:11.265957 15037 raft_consensus.cc:2781] T ccc0fe3d4a234214beec3dd0e3e95344 P df3980e9202d4bb586c7c6b817158342 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:09:11.266091 15037 raft_consensus.cc:686] T ccc0fe3d4a234214beec3dd0e3e95344 P df3980e9202d4bb586c7c6b817158342 [term 1 LEADER]: Becoming Leader. State: Replica: df3980e9202d4bb586c7c6b817158342, State: Running, Role: LEADER
I20260810 04:09:11.266194 15037 consensus_queue.cc:237] T ccc0fe3d4a234214beec3dd0e3e95344 P df3980e9202d4bb586c7c6b817158342 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2195f4b573b84fda85724f693eda2a36" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 46777 } } peers { permanent_uuid: "cf99f2b9d85949f8ae4e5bbe8021050b" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 34627 } } peers { permanent_uuid: "df3980e9202d4bb586c7c6b817158342" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 39207 } }
I20260810 04:09:11.267077 14509 catalog_manager.cc:5495] T ccc0fe3d4a234214beec3dd0e3e95344 P df3980e9202d4bb586c7c6b817158342 reported cstate change: term changed from 0 to 1, leader changed from <none> to df3980e9202d4bb586c7c6b817158342 (127.31.32.1). New cstate: current_term: 1 leader_uuid: "df3980e9202d4bb586c7c6b817158342" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2195f4b573b84fda85724f693eda2a36" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 46777 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cf99f2b9d85949f8ae4e5bbe8021050b" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 34627 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df3980e9202d4bb586c7c6b817158342" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 39207 } health_report { overall_health: HEALTHY } } }
I20260810 04:09:11.278898 15028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:11.276537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44340 (local address 127.31.32.1:39207)
0810 04:09:11.276590 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0810 04:09:11.276592 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:09:11.276604 (+ 12us) server_negotiation.cc:407] Connection header received
0810 04:09:11.276699 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:09:11.276700 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:09:11.276716 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:09:11.276789 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 04:09:11.276919 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:09:11.277384 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:09:11.278133 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:09:11.278326 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:09:11.278408 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:09:11.278579 (+ 171us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 04:09:11.278674 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 04:09:11.278765 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0810 04:09:11.278784 (+ 19us) server_negotiation.cc:299] Negotiation successful
0810 04:09:11.278805 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260810 04:09:11.280573 14817 raft_consensus.cc:1261] T ccc0fe3d4a234214beec3dd0e3e95344 P cf99f2b9d85949f8ae4e5bbe8021050b [term 1 FOLLOWER]: Refusing update from remote peer df3980e9202d4bb586c7c6b817158342: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:09:11.280583 14961 raft_consensus.cc:1261] T ccc0fe3d4a234214beec3dd0e3e95344 P 2195f4b573b84fda85724f693eda2a36 [term 1 FOLLOWER]: Refusing update from remote peer df3980e9202d4bb586c7c6b817158342: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:09:11.280905 15040 consensus_queue.cc:1035] T ccc0fe3d4a234214beec3dd0e3e95344 P df3980e9202d4bb586c7c6b817158342 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2195f4b573b84fda85724f693eda2a36" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 46777 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:09:11.281032 15037 consensus_queue.cc:1035] T ccc0fe3d4a234214beec3dd0e3e95344 P df3980e9202d4bb586c7c6b817158342 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf99f2b9d85949f8ae4e5bbe8021050b" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 34627 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:09:11.283583 15045 mvcc.cc:205] Tried to move back new op lower bound from 7316827960442683392 to 7316827960387239936. Current Snapshot: MvccSnapshot[applied={T|T < 7316827960442683392}]
I20260810 04:09:11.283717 15044 mvcc.cc:205] Tried to move back new op lower bound from 7316827960442683392 to 7316827960387239936. Current Snapshot: MvccSnapshot[applied={T|T < 7316827960442683392}]
I20260810 04:09:11.284492 15047 mvcc.cc:205] Tried to move back new op lower bound from 7316827960442683392 to 7316827960387239936. Current Snapshot: MvccSnapshot[applied={T|T < 7316827960442683392}]
I20260810 04:09:11.286000 14509 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50852:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 04:09:11.286091 14509 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50852:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 04:09:11.286809 14509 catalog_manager.cc:5782] T 00000000000000000000000000000000 P affc5cedaef749e2b7f7f15f09b0ecd0: Sending DeleteTablet for 3 replicas of tablet ccc0fe3d4a234214beec3dd0e3e95344
I20260810 04:09:11.287122 14940 tablet_service.cc:1514] Processing DeleteTablet for tablet ccc0fe3d4a234214beec3dd0e3e95344 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:09:11 UTC) from {username='kudu', principal='kudu/127.31.32.62@KRBTEST.COM'} at 127.0.0.1:49962
I20260810 04:09:11.287173 14797 tablet_service.cc:1514] Processing DeleteTablet for tablet ccc0fe3d4a234214beec3dd0e3e95344 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:09:11 UTC) from {username='kudu', principal='kudu/127.31.32.62@KRBTEST.COM'} at 127.0.0.1:54424
I20260810 04:09:11.287388 15056 tablet_replica.cc:330] T ccc0fe3d4a234214beec3dd0e3e95344 P cf99f2b9d85949f8ae4e5bbe8021050b: stopping tablet replica
I20260810 04:09:11.287396 14652 tablet_service.cc:1514] Processing DeleteTablet for tablet ccc0fe3d4a234214beec3dd0e3e95344 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:09:11 UTC) from {username='kudu', principal='kudu/127.31.32.62@KRBTEST.COM'} at 127.0.0.1:44328
I20260810 04:09:11.287493 15056 raft_consensus.cc:2229] T ccc0fe3d4a234214beec3dd0e3e95344 P cf99f2b9d85949f8ae4e5bbe8021050b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 04:09:11.287593 15056 raft_consensus.cc:2258] T ccc0fe3d4a234214beec3dd0e3e95344 P cf99f2b9d85949f8ae4e5bbe8021050b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:09:11.287604 31872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskws89Io/build/release/bin/kudu with pid 14583
I20260810 04:09:11.288048 15056 ts_tablet_manager.cc:1905] T ccc0fe3d4a234214beec3dd0e3e95344 P cf99f2b9d85949f8ae4e5bbe8021050b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:09:11.289513 15056 ts_tablet_manager.cc:1918] T ccc0fe3d4a234214beec3dd0e3e95344 P cf99f2b9d85949f8ae4e5bbe8021050b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 04:09:11.289583 15056 log.cc:1196] T ccc0fe3d4a234214beec3dd0e3e95344 P cf99f2b9d85949f8ae4e5bbe8021050b: Deleting WAL directory at /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/ts-1/wal/wals/ccc0fe3d4a234214beec3dd0e3e95344
I20260810 04:09:11.289819 15056 ts_tablet_manager.cc:1939] T ccc0fe3d4a234214beec3dd0e3e95344 P cf99f2b9d85949f8ae4e5bbe8021050b: Deleting consensus metadata
I20260810 04:09:11.290313 14491 catalog_manager.cc:4841] TS cf99f2b9d85949f8ae4e5bbe8021050b (127.31.32.2:34627): tablet ccc0fe3d4a234214beec3dd0e3e95344 (table test-table [id=cfc46bc4c2b54273aa6f612bb39b5cf8]) successfully deleted
W20260810 04:09:11.296104 14495 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.32.1:39207 (error 108)
I20260810 04:09:11.296162 15055 tablet_replica.cc:330] T ccc0fe3d4a234214beec3dd0e3e95344 P 2195f4b573b84fda85724f693eda2a36: stopping tablet replica
I20260810 04:09:11.296263 15055 raft_consensus.cc:2229] T ccc0fe3d4a234214beec3dd0e3e95344 P 2195f4b573b84fda85724f693eda2a36 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 04:09:11.296353 31872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskws89Io/build/release/bin/kudu with pid 14727
I20260810 04:09:11.296389 15055 raft_consensus.cc:2258] T ccc0fe3d4a234214beec3dd0e3e95344 P 2195f4b573b84fda85724f693eda2a36 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:09:11.298243 15027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:11.298102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.1:39207 (local address 127.0.0.1:44344)
0810 04:09:11.298167 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0810 04:09:11.298206 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.1:39207: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260810 04:09:11.298429 14495 catalog_manager.cc:4584] TS df3980e9202d4bb586c7c6b817158342 (127.31.32.1:39207): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ccc0fe3d4a234214beec3dd0e3e95344: Network error: Client connection negotiation failed: client connection to 127.31.32.1:39207: connect: Connection refused (error 111)
I20260810 04:09:11.298995 15055 ts_tablet_manager.cc:1905] T ccc0fe3d4a234214beec3dd0e3e95344 P 2195f4b573b84fda85724f693eda2a36: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:09:11.300472 15055 ts_tablet_manager.cc:1918] T ccc0fe3d4a234214beec3dd0e3e95344 P 2195f4b573b84fda85724f693eda2a36: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 04:09:11.300540 15055 log.cc:1196] T ccc0fe3d4a234214beec3dd0e3e95344 P 2195f4b573b84fda85724f693eda2a36: Deleting WAL directory at /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/ts-2/wal/wals/ccc0fe3d4a234214beec3dd0e3e95344
I20260810 04:09:11.300832 15055 ts_tablet_manager.cc:1939] T ccc0fe3d4a234214beec3dd0e3e95344 P 2195f4b573b84fda85724f693eda2a36: Deleting consensus metadata
I20260810 04:09:11.301244 14495 catalog_manager.cc:4841] TS 2195f4b573b84fda85724f693eda2a36 (127.31.32.3:46777): tablet ccc0fe3d4a234214beec3dd0e3e95344 (table test-table [id=cfc46bc4c2b54273aa6f612bb39b5cf8]) successfully deleted
I20260810 04:09:11.302289 31872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskws89Io/build/release/bin/kudu with pid 14874
I20260810 04:09:11.308739 31872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskws89Io/build/release/bin/kudu with pid 14474
I20260810 04:09:11.315380 31872 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskws89Io/build/release/bin/kudu
/tmp/dist-test-taskws89Io/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.32.62:44907
--webserver_interface=127.31.32.62
--webserver_port=44181
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.62.keytab
--principal=kudu/127.31.32.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.32.20:34365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.32.62:44907
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:09:11.395622 15059 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:09:11.395807 15059 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:09:11.395830 15059 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:09:11.396989 15059 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:09:11.397030 15059 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:09:11.397043 15059 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:09:11.397055 15059 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:09:11.397068 15059 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:09:11.398504 15059 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.32.20:34365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.31.32.62:44907
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.62.keytab
--principal=kudu/127.31.32.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.32.62:44907
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.32.62
--webserver_port=44181
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15059
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:09:11.398758 15059 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:09:11.398985 15059 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:09:11.407575 15066 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:09:11.407625 15069 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:09:11.407691 15065 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:09:11.407888 15059 server_base.cc:1034] running on GCE node
I20260810 04:09:11.408300 15059 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:09:11.408603 15059 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:09:11.409771 15059 hybrid_clock.cc:648] HybridClock initialized: now 1786334951409761 us; error 36 us; skew 500 ppm
Aug 10 04:09:11 dist-test-slave-8w9v krb5kdc[14461](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334951, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:09:11.415618 15059 init.cc:377] Logged in from keytab as kudu/127.31.32.62@KRBTEST.COM (short username kudu)
I20260810 04:09:11.416066 15059 webserver.cc:460] Webserver started at http://127.31.32.62:44181/ using document root <none> and password file <none>
I20260810 04:09:11.416303 15059 fs_manager.cc:359] Metadata directory not provided
I20260810 04:09:11.416359 15059 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:09:11.417626 15059 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:09:11.418274 15076 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:09:11.418588 15059 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:09:11.418684 15059 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/master-0/data,/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/master-0/wal
uuid: "affc5cedaef749e2b7f7f15f09b0ecd0"
format_stamp: "Formatted at 2026-08-10 04:09:10 on dist-test-slave-8w9v"
server_key: "c5e671ecb9c5e9286496d0ca6022cf05"
server_key_iv: "e2e14dd5d26b733c9145fbb549cb1f30"
server_key_version: "encryptionkey@0"
I20260810 04:09:11.419116 15059 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:09:11.433753 15059 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:09:11.435628 15059 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:09:11.435801 15059 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:09:11.440092 15059 rpc_server.cc:307] RPC server started. Bound to: 127.31.32.62:44907
I20260810 04:09:11.440165 15134 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.32.62:44907 every 8 connection(s)
I20260810 04:09:11.440595 15059 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/master-0/data/info.pb
I20260810 04:09:11.440801 31872 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskws89Io/build/release/bin/kudu as pid 15059
I20260810 04:09:11.441143 31872 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskws89Io/build/release/bin/kudu
/tmp/dist-test-taskws89Io/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.32.1:39207
--local_ip_for_outbound_sockets=127.31.32.1
--tserver_master_addrs=127.31.32.62:44907
--webserver_port=45699
--webserver_interface=127.31.32.1
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.1.keytab
--principal=kudu/127.31.32.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.32.20:34365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 04:09:11.442386 15135 sys_catalog.cc:263] Verifying existing consensus state
I20260810 04:09:11.443122 15135 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P affc5cedaef749e2b7f7f15f09b0ecd0: Bootstrap starting.
I20260810 04:09:11.451769 15135 log.cc:824] T 00000000000000000000000000000000 P affc5cedaef749e2b7f7f15f09b0ecd0: Log is configured to *not* fsync() on all Append() calls
I20260810 04:09:11.453847 15135 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P affc5cedaef749e2b7f7f15f09b0ecd0: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 04:09:11.454166 15135 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P affc5cedaef749e2b7f7f15f09b0ecd0: Bootstrap complete.
I20260810 04:09:11.456678 15135 raft_consensus.cc:348] T 00000000000000000000000000000000 P affc5cedaef749e2b7f7f15f09b0ecd0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "affc5cedaef749e2b7f7f15f09b0ecd0" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 44907 } }
I20260810 04:09:11.456933 15135 raft_consensus.cc:729] T 00000000000000000000000000000000 P affc5cedaef749e2b7f7f15f09b0ecd0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: affc5cedaef749e2b7f7f15f09b0ecd0, State: Initialized, Role: FOLLOWER
I20260810 04:09:11.457115 15135 consensus_queue.cc:260] T 00000000000000000000000000000000 P affc5cedaef749e2b7f7f15f09b0ecd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "affc5cedaef749e2b7f7f15f09b0ecd0" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 44907 } }
I20260810 04:09:11.457204 15135 raft_consensus.cc:388] T 00000000000000000000000000000000 P affc5cedaef749e2b7f7f15f09b0ecd0 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 04:09:11.457264 15135 raft_consensus.cc:482] T 00000000000000000000000000000000 P affc5cedaef749e2b7f7f15f09b0ecd0 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 04:09:11.457312 15135 raft_consensus.cc:3037] T 00000000000000000000000000000000 P affc5cedaef749e2b7f7f15f09b0ecd0 [term 1 FOLLOWER]: Advancing to term 2
I20260810 04:09:11.458324 15135 raft_consensus.cc:504] T 00000000000000000000000000000000 P affc5cedaef749e2b7f7f15f09b0ecd0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "affc5cedaef749e2b7f7f15f09b0ecd0" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 44907 } }
I20260810 04:09:11.458442 15135 leader_election.cc:302] T 00000000000000000000000000000000 P affc5cedaef749e2b7f7f15f09b0ecd0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: affc5cedaef749e2b7f7f15f09b0ecd0; no voters:
I20260810 04:09:11.458621 15135 leader_election.cc:288] T 00000000000000000000000000000000 P affc5cedaef749e2b7f7f15f09b0ecd0 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260810 04:09:11.458678 15141 raft_consensus.cc:2781] T 00000000000000000000000000000000 P affc5cedaef749e2b7f7f15f09b0ecd0 [term 2 FOLLOWER]: Leader election won for term 2
I20260810 04:09:11.458841 15141 raft_consensus.cc:686] T 00000000000000000000000000000000 P affc5cedaef749e2b7f7f15f09b0ecd0 [term 2 LEADER]: Becoming Leader. State: Replica: affc5cedaef749e2b7f7f15f09b0ecd0, State: Running, Role: LEADER
I20260810 04:09:11.458916 15135 sys_catalog.cc:564] T 00000000000000000000000000000000 P affc5cedaef749e2b7f7f15f09b0ecd0 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:09:11.458950 15141 consensus_queue.cc:237] T 00000000000000000000000000000000 P affc5cedaef749e2b7f7f15f09b0ecd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "affc5cedaef749e2b7f7f15f09b0ecd0" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 44907 } }
I20260810 04:09:11.459172 15141 sys_catalog.cc:455] T 00000000000000000000000000000000 P affc5cedaef749e2b7f7f15f09b0ecd0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader affc5cedaef749e2b7f7f15f09b0ecd0. Latest consensus state: current_term: 2 leader_uuid: "affc5cedaef749e2b7f7f15f09b0ecd0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "affc5cedaef749e2b7f7f15f09b0ecd0" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 44907 } } }
I20260810 04:09:11.459233 15141 sys_catalog.cc:458] T 00000000000000000000000000000000 P affc5cedaef749e2b7f7f15f09b0ecd0 [sys.catalog]: This master's current role is: LEADER
I20260810 04:09:11.459198 15143 sys_catalog.cc:455] T 00000000000000000000000000000000 P affc5cedaef749e2b7f7f15f09b0ecd0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "affc5cedaef749e2b7f7f15f09b0ecd0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "affc5cedaef749e2b7f7f15f09b0ecd0" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 44907 } } }
I20260810 04:09:11.459440 15143 sys_catalog.cc:458] T 00000000000000000000000000000000 P affc5cedaef749e2b7f7f15f09b0ecd0 [sys.catalog]: This master's current role is: LEADER
I20260810 04:09:11.460924 15158 catalog_manager.cc:1259] Loaded cluster ID: 3cb9a05ff41b4b4f9928b6cfef7d4d8e
I20260810 04:09:11.460983 15158 catalog_manager.cc:1552] T 00000000000000000000000000000000 P affc5cedaef749e2b7f7f15f09b0ecd0: loading cluster ID for follower catalog manager: success
W20260810 04:09:11.461212 15158 catalog_manager.cc:1576] T 00000000000000000000000000000000 P affc5cedaef749e2b7f7f15f09b0ecd0: acquiring CA information for follower catalog manager: Runtime error: could not decrypt private key with the password returned by the configured command: unable to load data from memory: error reading private key: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D07803A:asn1 encoding routines:ASN1_ITEM_EX_D2I:nested asn1 error:tasn_dec.c:386:Type=X509_ALGOR error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=algor, Type=X509_SIG
W20260810 04:09:11.461287 15158 catalog_manager.cc:873] Runtime error: could not decrypt private key with the password returned by the configured command: unable to load data from memory: error reading private key: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D07803A:asn1 encoding routines:ASN1_ITEM_EX_D2I:nested asn1 error:tasn_dec.c:386:Type=X509_ALGOR error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=algor, Type=X509_SIG: failed to prepare follower catalog manager, will retry
I20260810 04:09:11.461325 15145 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 04:09:11.461699 15145 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 04:09:11.461790 15145 catalog_manager.cc:1259] Loaded cluster ID: 3cb9a05ff41b4b4f9928b6cfef7d4d8e
I20260810 04:09:11.461830 15145 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 04:09:11.461989 15145 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260810 04:09:11.466202 15145 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260810 04:09:11.468628 15145 catalog_manager.cc:1504] Loading token signing keys...
I20260810 04:09:11.468848 15145 catalog_manager.cc:5879] T 00000000000000000000000000000000 P affc5cedaef749e2b7f7f15f09b0ecd0: Loaded TSK: 0
I20260810 04:09:11.469080 15145 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 04:09:11.524935 15137 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:09:11.525144 15137 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:09:11.525185 15137 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:09:11.526312 15137 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:09:11.526377 15137 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:09:11.526422 15137 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.32.1
I20260810 04:09:11.527850 15137 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.32.20:34365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.1.keytab
--principal=kudu/127.31.32.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.32.1:39207
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.32.1
--webserver_port=45699
--tserver_master_addrs=127.31.32.62:44907
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15137
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.32.1
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:09:11.528098 15137 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:09:11.528373 15137 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:09:11.533277 15165 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:09:11.533342 15168 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:09:11.533319 15166 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:09:11.533623 15137 server_base.cc:1034] running on GCE node
I20260810 04:09:11.533824 15137 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:09:11.534066 15137 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:09:11.535207 15137 hybrid_clock.cc:648] HybridClock initialized: now 1786334951535189 us; error 38 us; skew 500 ppm
Aug 10 04:09:11 dist-test-slave-8w9v krb5kdc[14461](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334951, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:09:11.537981 15137 init.cc:377] Logged in from keytab as kudu/127.31.32.1@KRBTEST.COM (short username kudu)
I20260810 04:09:11.538559 15137 webserver.cc:460] Webserver started at http://127.31.32.1:45699/ using document root <none> and password file <none>
I20260810 04:09:11.538830 15137 fs_manager.cc:359] Metadata directory not provided
I20260810 04:09:11.538918 15137 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:09:11.540773 15137 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260810 04:09:11.541591 15175 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:09:11.541787 15137 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:09:11.541891 15137 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/ts-0/data,/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/ts-0/wal
uuid: "df3980e9202d4bb586c7c6b817158342"
format_stamp: "Formatted at 2026-08-10 04:09:10 on dist-test-slave-8w9v"
server_key: "b92f55ed48a4450688ecb93317a5422c"
server_key_iv: "f96d477058c33bdca2dbf94b4555b30f"
server_key_version: "encryptionkey@0"
I20260810 04:09:11.542359 15137 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:09:11.562330 15137 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:09:11.564687 15137 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:09:11.564852 15137 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:09:11.565109 15137 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:09:11.565586 15182 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260810 04:09:11.567006 15137 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260810 04:09:11.567118 15137 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260810 04:09:11.567171 15137 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260810 04:09:11.568084 15137 ts_tablet_manager.cc:610] Registered 1 tablets
I20260810 04:09:11.568166 15137 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.001s sys 0.000s
I20260810 04:09:11.568209 15182 tablet_bootstrap.cc:492] T ccc0fe3d4a234214beec3dd0e3e95344 P df3980e9202d4bb586c7c6b817158342: Bootstrap starting.
I20260810 04:09:11.576594 15137 rpc_server.cc:307] RPC server started. Bound to: 127.31.32.1:39207
I20260810 04:09:11.576767 15302 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.32.1:39207 every 8 connection(s)
I20260810 04:09:11.577047 15137 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/ts-0/data/info.pb
I20260810 04:09:11.577345 31872 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskws89Io/build/release/bin/kudu as pid 15137
I20260810 04:09:11.577663 31872 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskws89Io/build/release/bin/kudu
/tmp/dist-test-taskws89Io/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.32.2:34627
--local_ip_for_outbound_sockets=127.31.32.2
--tserver_master_addrs=127.31.32.62:44907
--webserver_port=42345
--webserver_interface=127.31.32.2
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.2.keytab
--principal=kudu/127.31.32.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.32.20:34365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 04:09:11.578473 15182 log.cc:824] T ccc0fe3d4a234214beec3dd0e3e95344 P df3980e9202d4bb586c7c6b817158342: Log is configured to *not* fsync() on all Append() calls
I20260810 04:09:11.581825 15182 tablet_bootstrap.cc:492] T ccc0fe3d4a234214beec3dd0e3e95344 P df3980e9202d4bb586c7c6b817158342: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 04:09:11.582139 15182 tablet_bootstrap.cc:492] T ccc0fe3d4a234214beec3dd0e3e95344 P df3980e9202d4bb586c7c6b817158342: Bootstrap complete.
I20260810 04:09:11.583485 15182 ts_tablet_manager.cc:1397] T ccc0fe3d4a234214beec3dd0e3e95344 P df3980e9202d4bb586c7c6b817158342: Time spent bootstrapping tablet: real 0.015s user 0.009s sys 0.004s
I20260810 04:09:11.585273 15182 raft_consensus.cc:348] T ccc0fe3d4a234214beec3dd0e3e95344 P df3980e9202d4bb586c7c6b817158342 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2195f4b573b84fda85724f693eda2a36" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 46777 } } peers { permanent_uuid: "cf99f2b9d85949f8ae4e5bbe8021050b" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 34627 } } peers { permanent_uuid: "df3980e9202d4bb586c7c6b817158342" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 39207 } }
I20260810 04:09:11.585695 15182 raft_consensus.cc:729] T ccc0fe3d4a234214beec3dd0e3e95344 P df3980e9202d4bb586c7c6b817158342 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: df3980e9202d4bb586c7c6b817158342, State: Initialized, Role: FOLLOWER
I20260810 04:09:11.585868 15182 consensus_queue.cc:260] T ccc0fe3d4a234214beec3dd0e3e95344 P df3980e9202d4bb586c7c6b817158342 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2195f4b573b84fda85724f693eda2a36" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 46777 } } peers { permanent_uuid: "cf99f2b9d85949f8ae4e5bbe8021050b" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 34627 } } peers { permanent_uuid: "df3980e9202d4bb586c7c6b817158342" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 39207 } }
I20260810 04:09:11.586159 15182 ts_tablet_manager.cc:1428] T ccc0fe3d4a234214beec3dd0e3e95344 P df3980e9202d4bb586c7c6b817158342: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
Aug 10 04:09:11 dist-test-slave-8w9v krb5kdc[14461](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334951, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.1@KRBTEST.COM for kudu/127.31.32.62@KRBTEST.COM
I20260810 04:09:11.595280 15308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:11.579290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:44907 (local address 127.31.32.1:48409)
0810 04:09:11.582124 (+ 2834us) negotiation.cc:107] Waiting for socket to connect
0810 04:09:11.582170 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0810 04:09:11.582891 (+ 721us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:09:11.583685 (+ 794us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:09:11.583689 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:09:11.584073 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:09:11.584457 (+ 384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:09:11.584464 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:09:11.586378 (+ 1914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:09:11.586379 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:09:11.587730 (+ 1351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:09:11.587733 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:09:11.588593 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:09:11.588594 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:09:11.588743 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:09:11.589403 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:09:11.589414 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:09:11.591277 (+ 1863us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:09:11.593569 (+ 2292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:09:11.593576 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:09:11.593578 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:09:11.593866 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:09:11.594153 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:09:11.594154 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:09:11.594154 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:09:11.594275 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:09:11.594682 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:09:11.594686 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:09:11.594823 (+ 137us) client_negotiation.cc:769] Sending connection context
0810 04:09:11.594926 (+ 103us) client_negotiation.cc:240] Negotiation successful
0810 04:09:11.595074 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2772,"thread_start_us":96,"threads_started":1}
I20260810 04:09:11.595345 15309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:11.579644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.1:48409 (local address 127.31.32.62:44907)
0810 04:09:11.579822 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0810 04:09:11.579842 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0810 04:09:11.583064 (+ 3222us) server_negotiation.cc:407] Connection header received
0810 04:09:11.583219 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:09:11.583226 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:09:11.583303 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:09:11.583476 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:09:11.584611 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:09:11.586281 (+ 1670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:09:11.587836 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:09:11.588493 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:09:11.588612 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:09:11.591391 (+ 2779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:09:11.591395 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:09:11.591395 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:09:11.591408 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:09:11.593469 (+ 2061us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:09:11.593947 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:09:11.593950 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:09:11.593950 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:09:11.594081 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:09:11.594342 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:09:11.594343 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:09:11.594343 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:09:11.594607 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:09:11.594696 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0810 04:09:11.594998 (+ 302us) server_negotiation.cc:299] Negotiation successful
0810 04:09:11.595134 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":79,"threads_started":1}
I20260810 04:09:11.596493 15303 heartbeater.cc:347] Connected to a master server at 127.31.32.62:44907
I20260810 04:09:11.596596 15303 heartbeater.cc:464] Registering TS with master...
I20260810 04:09:11.596768 15303 heartbeater.cc:511] Master 127.31.32.62:44907 requested a full tablet report, sending...
I20260810 04:09:11.597550 15096 ts_manager.cc:194] Registered new tserver with Master: df3980e9202d4bb586c7c6b817158342 (127.31.32.1:39207)
I20260810 04:09:11.598716 15096 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.32.1@KRBTEST.COM'} at 127.31.32.1:48409
I20260810 04:09:11.599658 15303 heartbeater.cc:503] Master 127.31.32.62:44907 was elected leader, sending a full tablet report...
I20260810 04:09:11.606642 15311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:11.598135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.1:39207 (local address 127.0.0.1:44360)
0810 04:09:11.599059 (+ 924us) negotiation.cc:107] Waiting for socket to connect
0810 04:09:11.599106 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0810 04:09:11.599936 (+ 830us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:09:11.602968 (+ 3032us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:09:11.602971 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:09:11.602974 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:09:11.603033 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:09:11.603035 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:09:11.604313 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:09:11.604314 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:09:11.605485 (+ 1171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:09:11.605488 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:09:11.606352 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:09:11.606353 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:09:11.606435 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:09:11.606448 (+ 13us) client_negotiation.cc:769] Sending connection context
0810 04:09:11.606519 (+ 71us) client_negotiation.cc:240] Negotiation successful
0810 04:09:11.606569 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":894,"thread_start_us":502,"threads_started":1}
I20260810 04:09:11.606664 15312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:11.598900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44360 (local address 127.31.32.1:39207)
0810 04:09:11.602772 (+ 3872us) server_negotiation.cc:206] Beginning negotiation
0810 04:09:11.602777 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 04:09:11.602788 (+ 11us) server_negotiation.cc:407] Connection header received
0810 04:09:11.602824 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:09:11.602826 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:09:11.602842 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:09:11.602943 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:09:11.603111 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:09:11.604217 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:09:11.605570 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:09:11.606258 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:09:11.606336 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:09:11.606384 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0810 04:09:11.606543 (+ 159us) server_negotiation.cc:299] Negotiation successful
0810 04:09:11.606583 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3836,"thread_start_us":75,"threads_started":1}
I20260810 04:09:11.607237 15210 tablet_service.cc:1514] Processing DeleteTablet for tablet ccc0fe3d4a234214beec3dd0e3e95344 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:09:11 UTC) from {username='kudu', principal='kudu/127.31.32.62@KRBTEST.COM'} at 127.0.0.1:44360
I20260810 04:09:11.607807 15313 tablet_replica.cc:330] stopping tablet replica
I20260810 04:09:11.607913 15313 raft_consensus.cc:2229] T ccc0fe3d4a234214beec3dd0e3e95344 P df3980e9202d4bb586c7c6b817158342 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 04:09:11.607980 15313 raft_consensus.cc:2258] T ccc0fe3d4a234214beec3dd0e3e95344 P df3980e9202d4bb586c7c6b817158342 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:09:11.608340 15313 ts_tablet_manager.cc:1905] T ccc0fe3d4a234214beec3dd0e3e95344 P df3980e9202d4bb586c7c6b817158342: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:09:11.610597 15313 ts_tablet_manager.cc:1918] T ccc0fe3d4a234214beec3dd0e3e95344 P df3980e9202d4bb586c7c6b817158342: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 04:09:11.610682 15313 log.cc:1196] T ccc0fe3d4a234214beec3dd0e3e95344 P df3980e9202d4bb586c7c6b817158342: Deleting WAL directory at /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/ts-0/wal/wals/ccc0fe3d4a234214beec3dd0e3e95344
I20260810 04:09:11.610946 15313 ts_tablet_manager.cc:1939] T ccc0fe3d4a234214beec3dd0e3e95344 P df3980e9202d4bb586c7c6b817158342: Deleting consensus metadata
I20260810 04:09:11.611454 15080 catalog_manager.cc:4841] TS df3980e9202d4bb586c7c6b817158342 (127.31.32.1:39207): tablet ccc0fe3d4a234214beec3dd0e3e95344 (table test-table [id=cfc46bc4c2b54273aa6f612bb39b5cf8]) successfully deleted
W20260810 04:09:11.665210 15307 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:09:11.665410 15307 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:09:11.665444 15307 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:09:11.666501 15307 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:09:11.666559 15307 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:09:11.666601 15307 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.32.2
I20260810 04:09:11.667933 15307 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.32.20:34365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.2.keytab
--principal=kudu/127.31.32.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.32.2:34627
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.32.2
--webserver_port=42345
--tserver_master_addrs=127.31.32.62:44907
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15307
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.32.2
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:09:11.668182 15307 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:09:11.668394 15307 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 04:09:11.673499 15307 server_base.cc:1034] running on GCE node
W20260810 04:09:11.673449 15321 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:09:11.673446 15318 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:09:11.673439 15319 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:09:11.673866 15307 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:09:11.674106 15307 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:09:11.675264 15307 hybrid_clock.cc:648] HybridClock initialized: now 1786334951675259 us; error 36 us; skew 500 ppm
Aug 10 04:09:11 dist-test-slave-8w9v krb5kdc[14461](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334951, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:09:11.677692 15307 init.cc:377] Logged in from keytab as kudu/127.31.32.2@KRBTEST.COM (short username kudu)
I20260810 04:09:11.678061 15307 webserver.cc:460] Webserver started at http://127.31.32.2:42345/ using document root <none> and password file <none>
I20260810 04:09:11.678233 15307 fs_manager.cc:359] Metadata directory not provided
I20260810 04:09:11.678285 15307 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:09:11.679610 15307 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:09:11.680186 15329 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:09:11.680351 15307 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 04:09:11.680426 15307 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/ts-1/data,/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/ts-1/wal
uuid: "cf99f2b9d85949f8ae4e5bbe8021050b"
format_stamp: "Formatted at 2026-08-10 04:09:11 on dist-test-slave-8w9v"
server_key: "80e271ba77cc9983241470c2b1abf7a9"
server_key_iv: "945fbf70d98a4ebfbc2c8275c88b2169"
server_key_version: "encryptionkey@0"
I20260810 04:09:11.680727 15307 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:09:11.690703 15307 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:09:11.692485 15307 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:09:11.692589 15307 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:09:11.692749 15307 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:09:11.692975 15307 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:09:11.693004 15307 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:09:11.693023 15307 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:09:11.693037 15307 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:09:11.698452 15307 rpc_server.cc:307] RPC server started. Bound to: 127.31.32.2:34627
I20260810 04:09:11.698521 15452 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.32.2:34627 every 8 connection(s)
I20260810 04:09:11.698820 15307 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/ts-1/data/info.pb
I20260810 04:09:11.704152 31872 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskws89Io/build/release/bin/kudu as pid 15307
Aug 10 04:09:11 dist-test-slave-8w9v krb5kdc[14461](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334951, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.2@KRBTEST.COM for kudu/127.31.32.62@KRBTEST.COM
I20260810 04:09:11.704489 31872 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskws89Io/build/release/bin/kudu
/tmp/dist-test-taskws89Io/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.32.3:46777
--local_ip_for_outbound_sockets=127.31.32.3
--tserver_master_addrs=127.31.32.62:44907
--webserver_port=33751
--webserver_interface=127.31.32.3
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.3.keytab
--principal=kudu/127.31.32.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.32.20:34365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 04:09:11.707885 15309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:11.699479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.2:60603 (local address 127.31.32.62:44907)
0810 04:09:11.699549 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0810 04:09:11.699551 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:09:11.699718 (+ 167us) server_negotiation.cc:407] Connection header received
0810 04:09:11.700205 (+ 487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:09:11.700206 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:09:11.700225 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:09:11.700273 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:09:11.701099 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:09:11.701614 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:09:11.702527 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:09:11.702718 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:09:11.702766 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:09:11.704999 (+ 2233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:09:11.705000 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:09:11.705000 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:09:11.705007 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:09:11.706380 (+ 1373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:09:11.706864 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:09:11.706865 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:09:11.706865 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:09:11.706916 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:09:11.707252 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:09:11.707253 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:09:11.707253 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:09:11.707365 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:09:11.707413 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0810 04:09:11.707743 (+ 330us) server_negotiation.cc:299] Negotiation successful
0810 04:09:11.707812 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260810 04:09:11.708180 15456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:11.699447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:44907 (local address 127.31.32.2:60603)
0810 04:09:11.699611 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0810 04:09:11.699642 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0810 04:09:11.700071 (+ 429us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:09:11.700322 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:09:11.700325 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:09:11.700854 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:09:11.701024 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:09:11.701029 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:09:11.701691 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:09:11.701692 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:09:11.702461 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:09:11.702462 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:09:11.702791 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:09:11.702791 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:09:11.702881 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:09:11.703263 (+ 382us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:09:11.703271 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:09:11.704892 (+ 1621us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:09:11.706460 (+ 1568us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:09:11.706467 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:09:11.706469 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:09:11.706783 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:09:11.706977 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:09:11.706978 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:09:11.706979 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:09:11.707167 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:09:11.707427 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:09:11.707433 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:09:11.707606 (+ 173us) client_negotiation.cc:769] Sending connection context
0810 04:09:11.707715 (+ 109us) client_negotiation.cc:240] Negotiation successful
0810 04:09:11.707887 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":105,"threads_started":1}
I20260810 04:09:11.708700 15453 heartbeater.cc:347] Connected to a master server at 127.31.32.62:44907
I20260810 04:09:11.708811 15453 heartbeater.cc:464] Registering TS with master...
I20260810 04:09:11.709009 15453 heartbeater.cc:511] Master 127.31.32.62:44907 requested a full tablet report, sending...
I20260810 04:09:11.709415 15096 ts_manager.cc:194] Registered new tserver with Master: cf99f2b9d85949f8ae4e5bbe8021050b (127.31.32.2:34627)
I20260810 04:09:11.709774 15096 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.32.2@KRBTEST.COM'} at 127.31.32.2:60603
W20260810 04:09:11.789290 15457 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:09:11.789469 15457 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:09:11.789490 15457 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:09:11.790623 15457 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:09:11.790663 15457 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:09:11.790710 15457 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.32.3
I20260810 04:09:11.792204 15457 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.32.20:34365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.3.keytab
--principal=kudu/127.31.32.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.32.3:46777
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.32.3
--webserver_port=33751
--tserver_master_addrs=127.31.32.62:44907
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15457
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.32.3
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:09:11.792438 15457 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:09:11.792654 15457 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:09:11.797240 15465 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:09:11.797267 15462 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:09:11.797286 15463 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:09:11.797523 15457 server_base.cc:1034] running on GCE node
I20260810 04:09:11.797685 15457 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:09:11.797859 15457 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:09:11.799023 15457 hybrid_clock.cc:648] HybridClock initialized: now 1786334951799011 us; error 26 us; skew 500 ppm
Aug 10 04:09:11 dist-test-slave-8w9v krb5kdc[14461](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334951, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:09:11.801471 15457 init.cc:377] Logged in from keytab as kudu/127.31.32.3@KRBTEST.COM (short username kudu)
I20260810 04:09:11.801877 15457 webserver.cc:460] Webserver started at http://127.31.32.3:33751/ using document root <none> and password file <none>
I20260810 04:09:11.802106 15457 fs_manager.cc:359] Metadata directory not provided
I20260810 04:09:11.802160 15457 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:09:11.803439 15457 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260810 04:09:11.803982 15473 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:09:11.804170 15457 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 04:09:11.804275 15457 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/ts-2/data,/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/ts-2/wal
uuid: "2195f4b573b84fda85724f693eda2a36"
format_stamp: "Formatted at 2026-08-10 04:09:11 on dist-test-slave-8w9v"
server_key: "70cd799f0634144a7b3b59ed05934c23"
server_key_iv: "3673ef5a08d4dc00e0b0e6746192deda"
server_key_version: "encryptionkey@0"
I20260810 04:09:11.804625 15457 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:09:11.826728 15457 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:09:11.828464 15457 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:09:11.828620 15457 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:09:11.828826 15457 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:09:11.829188 15457 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:09:11.829238 15457 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:09:11.829276 15457 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:09:11.829303 15457 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:09:11.834913 15457 rpc_server.cc:307] RPC server started. Bound to: 127.31.32.3:46777
I20260810 04:09:11.834950 15602 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.32.3:46777 every 8 connection(s)
I20260810 04:09:11.835273 15457 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/ts-2/data/info.pb
I20260810 04:09:11.840723 31872 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskws89Io/build/release/bin/kudu as pid 15457
Aug 10 04:09:11 dist-test-slave-8w9v krb5kdc[14461](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334951, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.3@KRBTEST.COM for kudu/127.31.32.62@KRBTEST.COM
I20260810 04:09:11.844571 15309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:11.835960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.3:57031 (local address 127.31.32.62:44907)
0810 04:09:11.836025 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0810 04:09:11.836027 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:09:11.836273 (+ 246us) server_negotiation.cc:407] Connection header received
0810 04:09:11.836754 (+ 481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:09:11.836755 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:09:11.836767 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:09:11.836816 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:09:11.837756 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:09:11.838240 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:09:11.839275 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:09:11.839491 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:09:11.839576 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:09:11.841735 (+ 2159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:09:11.841737 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:09:11.841737 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:09:11.841745 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:09:11.843182 (+ 1437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:09:11.843581 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:09:11.843582 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:09:11.843582 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:09:11.843639 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:09:11.843907 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:09:11.843908 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:09:11.843909 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:09:11.844057 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:09:11.844113 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0810 04:09:11.844445 (+ 332us) server_negotiation.cc:299] Negotiation successful
0810 04:09:11.844511 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260810 04:09:11.844825 15605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:11.835944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:44907 (local address 127.31.32.3:57031)
0810 04:09:11.836109 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0810 04:09:11.836191 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0810 04:09:11.836656 (+ 465us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:09:11.836884 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:09:11.836887 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:09:11.837480 (+ 593us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:09:11.837654 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:09:11.837659 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:09:11.838331 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:09:11.838332 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:09:11.839183 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:09:11.839185 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:09:11.839589 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:09:11.839590 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:09:11.839708 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:09:11.840177 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:09:11.840184 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:09:11.841652 (+ 1468us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:09:11.843259 (+ 1607us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:09:11.843266 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:09:11.843268 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:09:11.843496 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:09:11.843701 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:09:11.843702 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:09:11.843703 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:09:11.843844 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:09:11.844142 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:09:11.844148 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:09:11.844302 (+ 154us) client_negotiation.cc:769] Sending connection context
0810 04:09:11.844428 (+ 126us) client_negotiation.cc:240] Negotiation successful
0810 04:09:11.844592 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":92,"threads_started":1}
I20260810 04:09:11.845278 15603 heartbeater.cc:347] Connected to a master server at 127.31.32.62:44907
I20260810 04:09:11.845382 15603 heartbeater.cc:464] Registering TS with master...
I20260810 04:09:11.845547 15603 heartbeater.cc:511] Master 127.31.32.62:44907 requested a full tablet report, sending...
I20260810 04:09:11.845906 15096 ts_manager.cc:194] Registered new tserver with Master: 2195f4b573b84fda85724f693eda2a36 (127.31.32.3:46777)
I20260810 04:09:11.846259 15096 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.32.3@KRBTEST.COM'} at 127.31.32.3:57031
I20260810 04:09:11.847555 15607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:11.841086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50854 (local address 127.31.32.62:44907)
0810 04:09:11.841223 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0810 04:09:11.841225 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:09:11.841262 (+ 37us) server_negotiation.cc:407] Connection header received
0810 04:09:11.841379 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:09:11.841380 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:09:11.841392 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:09:11.841448 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:09:11.842117 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:09:11.842598 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:09:11.843304 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:09:11.843490 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:09:11.843537 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:09:11.845046 (+ 1509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:09:11.845047 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:09:11.845048 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:09:11.845053 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:09:11.846560 (+ 1507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:09:11.846898 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:09:11.846899 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:09:11.846900 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:09:11.846962 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:09:11.847128 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:09:11.847129 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:09:11.847129 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:09:11.847224 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:09:11.847349 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0810 04:09:11.847398 (+ 49us) server_negotiation.cc:299] Negotiation successful
0810 04:09:11.847484 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":81,"threads_started":1}
I20260810 04:09:11.848258 31872 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 04:09:11.850452 15607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:11.848566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50868 (local address 127.31.32.62:44907)
0810 04:09:11.848606 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0810 04:09:11.848608 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:09:11.848688 (+ 80us) server_negotiation.cc:407] Connection header received
0810 04:09:11.848738 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:09:11.848739 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:09:11.848765 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:09:11.848821 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 04:09:11.848889 (+ 68us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:09:11.849291 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:09:11.849914 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:09:11.850074 (+ 160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:09:11.850118 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:09:11.850219 (+ 101us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 04:09:11.850290 (+ 71us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 04:09:11.850337 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0810 04:09:11.850364 (+ 27us) server_negotiation.cc:299] Negotiation successful
0810 04:09:11.850377 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260810 04:09:11.850709 15096 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:50868:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 04:09:11.852207 15096 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 04:09:11.854108 15210 tablet_service.cc:1467] Processing CreateTablet for tablet 58fb54d17b9943cf8f21be2778663c72 (DEFAULT_TABLE table=test-table [id=c5ec07266c0d4990b26f5d910c97be11]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:09:11.854296 15210 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58fb54d17b9943cf8f21be2778663c72. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:09:11.855760 15182 tablet_bootstrap.cc:492] T 58fb54d17b9943cf8f21be2778663c72 P df3980e9202d4bb586c7c6b817158342: Bootstrap starting.
I20260810 04:09:11.856102 15182 tablet_bootstrap.cc:654] T 58fb54d17b9943cf8f21be2778663c72 P df3980e9202d4bb586c7c6b817158342: Neither blocks nor log segments found. Creating new log.
I20260810 04:09:11.856650 15182 tablet_bootstrap.cc:492] T 58fb54d17b9943cf8f21be2778663c72 P df3980e9202d4bb586c7c6b817158342: No bootstrap required, opened a new log
I20260810 04:09:11.856738 15182 ts_tablet_manager.cc:1397] T 58fb54d17b9943cf8f21be2778663c72 P df3980e9202d4bb586c7c6b817158342: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260810 04:09:11.856856 15182 raft_consensus.cc:348] T 58fb54d17b9943cf8f21be2778663c72 P df3980e9202d4bb586c7c6b817158342 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2195f4b573b84fda85724f693eda2a36" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 46777 } } peers { permanent_uuid: "cf99f2b9d85949f8ae4e5bbe8021050b" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 34627 } } peers { permanent_uuid: "df3980e9202d4bb586c7c6b817158342" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 39207 } }
I20260810 04:09:11.856928 15182 raft_consensus.cc:374] T 58fb54d17b9943cf8f21be2778663c72 P df3980e9202d4bb586c7c6b817158342 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:09:11.856976 15182 raft_consensus.cc:729] T 58fb54d17b9943cf8f21be2778663c72 P df3980e9202d4bb586c7c6b817158342 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df3980e9202d4bb586c7c6b817158342, State: Initialized, Role: FOLLOWER
I20260810 04:09:11.857024 15182 consensus_queue.cc:260] T 58fb54d17b9943cf8f21be2778663c72 P df3980e9202d4bb586c7c6b817158342 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2195f4b573b84fda85724f693eda2a36" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 46777 } } peers { permanent_uuid: "cf99f2b9d85949f8ae4e5bbe8021050b" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 34627 } } peers { permanent_uuid: "df3980e9202d4bb586c7c6b817158342" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 39207 } }
I20260810 04:09:11.857116 15182 ts_tablet_manager.cc:1428] T 58fb54d17b9943cf8f21be2778663c72 P df3980e9202d4bb586c7c6b817158342: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260810 04:09:11.857602 15311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:11.853798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.3:46777 (local address 127.0.0.1:49968)
0810 04:09:11.853848 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0810 04:09:11.853858 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 04:09:11.853891 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:09:11.854254 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:09:11.854255 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:09:11.854257 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:09:11.854290 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:09:11.854291 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:09:11.855505 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:09:11.855506 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:09:11.856534 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:09:11.856536 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:09:11.857395 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:09:11.857396 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:09:11.857463 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:09:11.857466 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 04:09:11.857496 (+ 30us) client_negotiation.cc:240] Negotiation successful
0810 04:09:11.857535 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260810 04:09:11.857722 15609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:11.853854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.2:34627 (local address 127.0.0.1:54440)
0810 04:09:11.853988 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0810 04:09:11.853998 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 04:09:11.854035 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:09:11.854364 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:09:11.854365 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:09:11.854368 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:09:11.854405 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:09:11.854406 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:09:11.855310 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:09:11.855311 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:09:11.856439 (+ 1128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:09:11.856442 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:09:11.857466 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:09:11.857467 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:09:11.857533 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:09:11.857535 (+ 2us) client_negotiation.cc:769] Sending connection context
0810 04:09:11.857562 (+ 27us) client_negotiation.cc:240] Negotiation successful
0810 04:09:11.857592 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"mutex_wait_us":30,"thread_start_us":81,"threads_started":1}
I20260810 04:09:11.857867 15611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:11.854051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54440 (local address 127.31.32.2:34627)
0810 04:09:11.854210 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0810 04:09:11.854214 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 04:09:11.854223 (+ 9us) server_negotiation.cc:407] Connection header received
0810 04:09:11.854262 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:09:11.854264 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:09:11.854278 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:09:11.854351 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:09:11.854462 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:09:11.855219 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:09:11.856546 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:09:11.857385 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:09:11.857642 (+ 257us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:09:11.857716 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0810 04:09:11.857744 (+ 28us) server_negotiation.cc:299] Negotiation successful
0810 04:09:11.857787 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":80,"threads_started":1}
I20260810 04:09:11.857890 15610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:11.853988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49968 (local address 127.31.32.3:46777)
0810 04:09:11.854122 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0810 04:09:11.854126 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 04:09:11.854135 (+ 9us) server_negotiation.cc:407] Connection header received
0810 04:09:11.854165 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:09:11.854166 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:09:11.854179 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:09:11.854258 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:09:11.854489 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:09:11.855387 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:09:11.856611 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:09:11.857295 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:09:11.857715 (+ 420us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:09:11.857769 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0810 04:09:11.857791 (+ 22us) server_negotiation.cc:299] Negotiation successful
0810 04:09:11.857823 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":64,"threads_started":1}
I20260810 04:09:11.858497 15381 tablet_service.cc:1467] Processing CreateTablet for tablet 58fb54d17b9943cf8f21be2778663c72 (DEFAULT_TABLE table=test-table [id=c5ec07266c0d4990b26f5d910c97be11]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:09:11.858539 15528 tablet_service.cc:1467] Processing CreateTablet for tablet 58fb54d17b9943cf8f21be2778663c72 (DEFAULT_TABLE table=test-table [id=c5ec07266c0d4990b26f5d910c97be11]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:09:11.858832 15381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58fb54d17b9943cf8f21be2778663c72. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:09:11.858834 15528 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58fb54d17b9943cf8f21be2778663c72. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:09:11.862133 15613 tablet_bootstrap.cc:492] T 58fb54d17b9943cf8f21be2778663c72 P 2195f4b573b84fda85724f693eda2a36: Bootstrap starting.
I20260810 04:09:11.862726 15613 tablet_bootstrap.cc:654] T 58fb54d17b9943cf8f21be2778663c72 P 2195f4b573b84fda85724f693eda2a36: Neither blocks nor log segments found. Creating new log.
I20260810 04:09:11.863204 15613 log.cc:824] T 58fb54d17b9943cf8f21be2778663c72 P 2195f4b573b84fda85724f693eda2a36: Log is configured to *not* fsync() on all Append() calls
I20260810 04:09:11.863508 15614 tablet_bootstrap.cc:492] T 58fb54d17b9943cf8f21be2778663c72 P cf99f2b9d85949f8ae4e5bbe8021050b: Bootstrap starting.
I20260810 04:09:11.863889 15613 tablet_bootstrap.cc:492] T 58fb54d17b9943cf8f21be2778663c72 P 2195f4b573b84fda85724f693eda2a36: No bootstrap required, opened a new log
I20260810 04:09:11.863945 15613 ts_tablet_manager.cc:1397] T 58fb54d17b9943cf8f21be2778663c72 P 2195f4b573b84fda85724f693eda2a36: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 04:09:11.864078 15614 tablet_bootstrap.cc:654] T 58fb54d17b9943cf8f21be2778663c72 P cf99f2b9d85949f8ae4e5bbe8021050b: Neither blocks nor log segments found. Creating new log.
I20260810 04:09:11.864537 15614 log.cc:824] T 58fb54d17b9943cf8f21be2778663c72 P cf99f2b9d85949f8ae4e5bbe8021050b: Log is configured to *not* fsync() on all Append() calls
I20260810 04:09:11.865159 15614 tablet_bootstrap.cc:492] T 58fb54d17b9943cf8f21be2778663c72 P cf99f2b9d85949f8ae4e5bbe8021050b: No bootstrap required, opened a new log
I20260810 04:09:11.865237 15614 ts_tablet_manager.cc:1397] T 58fb54d17b9943cf8f21be2778663c72 P cf99f2b9d85949f8ae4e5bbe8021050b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 04:09:11.865404 15613 raft_consensus.cc:348] T 58fb54d17b9943cf8f21be2778663c72 P 2195f4b573b84fda85724f693eda2a36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2195f4b573b84fda85724f693eda2a36" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 46777 } } peers { permanent_uuid: "cf99f2b9d85949f8ae4e5bbe8021050b" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 34627 } } peers { permanent_uuid: "df3980e9202d4bb586c7c6b817158342" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 39207 } }
I20260810 04:09:11.865536 15613 raft_consensus.cc:374] T 58fb54d17b9943cf8f21be2778663c72 P 2195f4b573b84fda85724f693eda2a36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:09:11.865579 15613 raft_consensus.cc:729] T 58fb54d17b9943cf8f21be2778663c72 P 2195f4b573b84fda85724f693eda2a36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2195f4b573b84fda85724f693eda2a36, State: Initialized, Role: FOLLOWER
I20260810 04:09:11.865712 15613 consensus_queue.cc:260] T 58fb54d17b9943cf8f21be2778663c72 P 2195f4b573b84fda85724f693eda2a36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2195f4b573b84fda85724f693eda2a36" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 46777 } } peers { permanent_uuid: "cf99f2b9d85949f8ae4e5bbe8021050b" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 34627 } } peers { permanent_uuid: "df3980e9202d4bb586c7c6b817158342" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 39207 } }
I20260810 04:09:11.865998 15613 ts_tablet_manager.cc:1428] T 58fb54d17b9943cf8f21be2778663c72 P 2195f4b573b84fda85724f693eda2a36: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260810 04:09:11.866019 15603 heartbeater.cc:503] Master 127.31.32.62:44907 was elected leader, sending a full tablet report...
I20260810 04:09:11.866911 15614 raft_consensus.cc:348] T 58fb54d17b9943cf8f21be2778663c72 P cf99f2b9d85949f8ae4e5bbe8021050b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2195f4b573b84fda85724f693eda2a36" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 46777 } } peers { permanent_uuid: "cf99f2b9d85949f8ae4e5bbe8021050b" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 34627 } } peers { permanent_uuid: "df3980e9202d4bb586c7c6b817158342" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 39207 } }
I20260810 04:09:11.867038 15614 raft_consensus.cc:374] T 58fb54d17b9943cf8f21be2778663c72 P cf99f2b9d85949f8ae4e5bbe8021050b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:09:11.867079 15614 raft_consensus.cc:729] T 58fb54d17b9943cf8f21be2778663c72 P cf99f2b9d85949f8ae4e5bbe8021050b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf99f2b9d85949f8ae4e5bbe8021050b, State: Initialized, Role: FOLLOWER
I20260810 04:09:11.867220 15614 consensus_queue.cc:260] T 58fb54d17b9943cf8f21be2778663c72 P cf99f2b9d85949f8ae4e5bbe8021050b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2195f4b573b84fda85724f693eda2a36" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 46777 } } peers { permanent_uuid: "cf99f2b9d85949f8ae4e5bbe8021050b" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 34627 } } peers { permanent_uuid: "df3980e9202d4bb586c7c6b817158342" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 39207 } }
I20260810 04:09:11.867467 15614 ts_tablet_manager.cc:1428] T 58fb54d17b9943cf8f21be2778663c72 P cf99f2b9d85949f8ae4e5bbe8021050b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260810 04:09:11.867516 15453 heartbeater.cc:503] Master 127.31.32.62:44907 was elected leader, sending a full tablet report...
I20260810 04:09:11.870445 15618 raft_consensus.cc:482] T 58fb54d17b9943cf8f21be2778663c72 P cf99f2b9d85949f8ae4e5bbe8021050b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 04:09:11.870512 15618 raft_consensus.cc:504] T 58fb54d17b9943cf8f21be2778663c72 P cf99f2b9d85949f8ae4e5bbe8021050b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2195f4b573b84fda85724f693eda2a36" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 46777 } } peers { permanent_uuid: "cf99f2b9d85949f8ae4e5bbe8021050b" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 34627 } } peers { permanent_uuid: "df3980e9202d4bb586c7c6b817158342" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 39207 } }
I20260810 04:09:11.870745 15618 leader_election.cc:288] T 58fb54d17b9943cf8f21be2778663c72 P cf99f2b9d85949f8ae4e5bbe8021050b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2195f4b573b84fda85724f693eda2a36 (127.31.32.3:46777), df3980e9202d4bb586c7c6b817158342 (127.31.32.1:39207)
I20260810 04:09:11.873754 15610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:11.870967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.2:42603 (local address 127.31.32.3:46777)
0810 04:09:11.871026 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0810 04:09:11.871028 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:09:11.871220 (+ 192us) server_negotiation.cc:407] Connection header received
0810 04:09:11.871294 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:09:11.871295 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:09:11.871304 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:09:11.871358 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:09:11.871505 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:09:11.871933 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:09:11.873015 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:09:11.873376 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:09:11.873480 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:09:11.873489 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0810 04:09:11.873648 (+ 159us) server_negotiation.cc:299] Negotiation successful
0810 04:09:11.873682 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260810 04:09:11.873752 15312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:11.871101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.2:33661 (local address 127.31.32.1:39207)
0810 04:09:11.871162 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0810 04:09:11.871164 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:09:11.871244 (+ 80us) server_negotiation.cc:407] Connection header received
0810 04:09:11.871311 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:09:11.871312 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:09:11.871322 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:09:11.871374 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:09:11.871515 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:09:11.871955 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:09:11.873002 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:09:11.873383 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:09:11.873477 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:09:11.873487 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0810 04:09:11.873638 (+ 151us) server_negotiation.cc:299] Negotiation successful
0810 04:09:11.873675 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260810 04:09:11.873767 15456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:11.871019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.1:39207 (local address 127.31.32.2:33661)
0810 04:09:11.871175 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0810 04:09:11.871185 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 04:09:11.871249 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:09:11.871385 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:09:11.871386 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:09:11.871387 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:09:11.871441 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:09:11.871442 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:09:11.872031 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:09:11.872032 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:09:11.872915 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:09:11.872917 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:09:11.873477 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:09:11.873478 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:09:11.873550 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:09:11.873554 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 04:09:11.873630 (+ 76us) client_negotiation.cc:240] Negotiation successful
0810 04:09:11.873675 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"mutex_wait_us":31,"thread_start_us":90,"threads_started":1}
I20260810 04:09:11.873775 15619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:11.870920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.3:46777 (local address 127.31.32.2:42603)
0810 04:09:11.871164 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0810 04:09:11.871175 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 04:09:11.871236 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:09:11.871362 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:09:11.871363 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:09:11.871364 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:09:11.871419 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:09:11.871420 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:09:11.872025 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:09:11.872026 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:09:11.872916 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:09:11.872918 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:09:11.873496 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:09:11.873497 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:09:11.873571 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:09:11.873574 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 04:09:11.873631 (+ 57us) client_negotiation.cc:240] Negotiation successful
0810 04:09:11.873676 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"mutex_wait_us":30}
I20260810 04:09:11.874071 15250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58fb54d17b9943cf8f21be2778663c72" candidate_uuid: "cf99f2b9d85949f8ae4e5bbe8021050b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df3980e9202d4bb586c7c6b817158342" is_pre_election: true
I20260810 04:09:11.874207 15250 raft_consensus.cc:2454] T 58fb54d17b9943cf8f21be2778663c72 P df3980e9202d4bb586c7c6b817158342 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf99f2b9d85949f8ae4e5bbe8021050b in term 0.
I20260810 04:09:11.874431 15333 leader_election.cc:302] T 58fb54d17b9943cf8f21be2778663c72 P cf99f2b9d85949f8ae4e5bbe8021050b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cf99f2b9d85949f8ae4e5bbe8021050b, df3980e9202d4bb586c7c6b817158342; no voters:
I20260810 04:09:11.874454 15551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58fb54d17b9943cf8f21be2778663c72" candidate_uuid: "cf99f2b9d85949f8ae4e5bbe8021050b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2195f4b573b84fda85724f693eda2a36" is_pre_election: true
I20260810 04:09:11.874552 15618 raft_consensus.cc:2781] T 58fb54d17b9943cf8f21be2778663c72 P cf99f2b9d85949f8ae4e5bbe8021050b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 04:09:11.874615 15618 raft_consensus.cc:482] T 58fb54d17b9943cf8f21be2778663c72 P cf99f2b9d85949f8ae4e5bbe8021050b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 04:09:11.874610 15551 raft_consensus.cc:2454] T 58fb54d17b9943cf8f21be2778663c72 P 2195f4b573b84fda85724f693eda2a36 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf99f2b9d85949f8ae4e5bbe8021050b in term 0.
I20260810 04:09:11.874650 15618 raft_consensus.cc:3037] T 58fb54d17b9943cf8f21be2778663c72 P cf99f2b9d85949f8ae4e5bbe8021050b [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:09:11.875416 15618 raft_consensus.cc:504] T 58fb54d17b9943cf8f21be2778663c72 P cf99f2b9d85949f8ae4e5bbe8021050b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2195f4b573b84fda85724f693eda2a36" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 46777 } } peers { permanent_uuid: "cf99f2b9d85949f8ae4e5bbe8021050b" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 34627 } } peers { permanent_uuid: "df3980e9202d4bb586c7c6b817158342" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 39207 } }
I20260810 04:09:11.875543 15618 leader_election.cc:288] T 58fb54d17b9943cf8f21be2778663c72 P cf99f2b9d85949f8ae4e5bbe8021050b [CANDIDATE]: Term 1 election: Requested vote from peers 2195f4b573b84fda85724f693eda2a36 (127.31.32.3:46777), df3980e9202d4bb586c7c6b817158342 (127.31.32.1:39207)
I20260810 04:09:11.875689 15551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58fb54d17b9943cf8f21be2778663c72" candidate_uuid: "cf99f2b9d85949f8ae4e5bbe8021050b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2195f4b573b84fda85724f693eda2a36"
I20260810 04:09:11.875721 15250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58fb54d17b9943cf8f21be2778663c72" candidate_uuid: "cf99f2b9d85949f8ae4e5bbe8021050b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df3980e9202d4bb586c7c6b817158342"
I20260810 04:09:11.875768 15551 raft_consensus.cc:3037] T 58fb54d17b9943cf8f21be2778663c72 P 2195f4b573b84fda85724f693eda2a36 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:09:11.875785 15250 raft_consensus.cc:3037] T 58fb54d17b9943cf8f21be2778663c72 P df3980e9202d4bb586c7c6b817158342 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:09:11.876649 15551 raft_consensus.cc:2454] T 58fb54d17b9943cf8f21be2778663c72 P 2195f4b573b84fda85724f693eda2a36 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf99f2b9d85949f8ae4e5bbe8021050b in term 1.
I20260810 04:09:11.876822 15333 leader_election.cc:302] T 58fb54d17b9943cf8f21be2778663c72 P cf99f2b9d85949f8ae4e5bbe8021050b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2195f4b573b84fda85724f693eda2a36, cf99f2b9d85949f8ae4e5bbe8021050b; no voters:
I20260810 04:09:11.876920 15618 raft_consensus.cc:2781] T 58fb54d17b9943cf8f21be2778663c72 P cf99f2b9d85949f8ae4e5bbe8021050b [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:09:11.877045 15250 raft_consensus.cc:2454] T 58fb54d17b9943cf8f21be2778663c72 P df3980e9202d4bb586c7c6b817158342 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf99f2b9d85949f8ae4e5bbe8021050b in term 1.
I20260810 04:09:11.877068 15618 raft_consensus.cc:686] T 58fb54d17b9943cf8f21be2778663c72 P cf99f2b9d85949f8ae4e5bbe8021050b [term 1 LEADER]: Becoming Leader. State: Replica: cf99f2b9d85949f8ae4e5bbe8021050b, State: Running, Role: LEADER
I20260810 04:09:11.877146 15618 consensus_queue.cc:237] T 58fb54d17b9943cf8f21be2778663c72 P cf99f2b9d85949f8ae4e5bbe8021050b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2195f4b573b84fda85724f693eda2a36" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 46777 } } peers { permanent_uuid: "cf99f2b9d85949f8ae4e5bbe8021050b" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 34627 } } peers { permanent_uuid: "df3980e9202d4bb586c7c6b817158342" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 39207 } }
I20260810 04:09:11.877831 15096 catalog_manager.cc:5495] T 58fb54d17b9943cf8f21be2778663c72 P cf99f2b9d85949f8ae4e5bbe8021050b reported cstate change: term changed from 0 to 1, leader changed from <none> to cf99f2b9d85949f8ae4e5bbe8021050b (127.31.32.2). New cstate: current_term: 1 leader_uuid: "cf99f2b9d85949f8ae4e5bbe8021050b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2195f4b573b84fda85724f693eda2a36" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 46777 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cf99f2b9d85949f8ae4e5bbe8021050b" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 34627 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df3980e9202d4bb586c7c6b817158342" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 39207 } health_report { overall_health: UNKNOWN } } }
I20260810 04:09:11.885399 15611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:11.883068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54450 (local address 127.31.32.2:34627)
0810 04:09:11.883123 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0810 04:09:11.883125 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:09:11.883133 (+ 8us) server_negotiation.cc:407] Connection header received
0810 04:09:11.883194 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:09:11.883195 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:09:11.883216 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:09:11.883298 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 04:09:11.883432 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:09:11.883881 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:09:11.884658 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:09:11.884836 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:09:11.884912 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:09:11.885094 (+ 182us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 04:09:11.885179 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 04:09:11.885255 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0810 04:09:11.885303 (+ 48us) server_negotiation.cc:299] Negotiation successful
0810 04:09:11.885321 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260810 04:09:11.886850 15551 raft_consensus.cc:1261] T 58fb54d17b9943cf8f21be2778663c72 P 2195f4b573b84fda85724f693eda2a36 [term 1 FOLLOWER]: Refusing update from remote peer cf99f2b9d85949f8ae4e5bbe8021050b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:09:11.886855 15250 raft_consensus.cc:1261] T 58fb54d17b9943cf8f21be2778663c72 P df3980e9202d4bb586c7c6b817158342 [term 1 FOLLOWER]: Refusing update from remote peer cf99f2b9d85949f8ae4e5bbe8021050b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:09:11.887151 15618 consensus_queue.cc:1035] T 58fb54d17b9943cf8f21be2778663c72 P cf99f2b9d85949f8ae4e5bbe8021050b [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3980e9202d4bb586c7c6b817158342" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 39207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:09:11.887271 15620 consensus_queue.cc:1035] T 58fb54d17b9943cf8f21be2778663c72 P cf99f2b9d85949f8ae4e5bbe8021050b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2195f4b573b84fda85724f693eda2a36" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 46777 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:09:11.889297 15624 mvcc.cc:205] Tried to move back new op lower bound from 7316827962926387200 to 7316827962889445376. Current Snapshot: MvccSnapshot[applied={T|T < 7316827962926387200}]
I20260810 04:09:11.889865 15623 mvcc.cc:205] Tried to move back new op lower bound from 7316827962926387200 to 7316827962889445376. Current Snapshot: MvccSnapshot[applied={T|T < 7316827962926387200 or (T in {7316827962926387200})}]
I20260810 04:09:11.890360 15625 mvcc.cc:205] Tried to move back new op lower bound from 7316827962926387200 to 7316827962889445376. Current Snapshot: MvccSnapshot[applied={T|T < 7316827962926387200}]
I20260810 04:09:11.891397 15093 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:50868:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 04:09:11.891489 15093 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:50868:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 04:09:11.892253 15093 catalog_manager.cc:5782] T 00000000000000000000000000000000 P affc5cedaef749e2b7f7f15f09b0ecd0: Sending DeleteTablet for 3 replicas of tablet 58fb54d17b9943cf8f21be2778663c72
I20260810 04:09:11.892566 15381 tablet_service.cc:1514] Processing DeleteTablet for tablet 58fb54d17b9943cf8f21be2778663c72 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:09:11 UTC) from {username='kudu', principal='kudu/127.31.32.62@KRBTEST.COM'} at 127.0.0.1:54440
I20260810 04:09:11.892604 31872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskws89Io/build/release/bin/kudu with pid 15137
I20260810 04:09:11.892650 15528 tablet_service.cc:1514] Processing DeleteTablet for tablet 58fb54d17b9943cf8f21be2778663c72 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:09:11 UTC) from {username='kudu', principal='kudu/127.31.32.62@KRBTEST.COM'} at 127.0.0.1:49968
I20260810 04:09:11.892834 15634 tablet_replica.cc:330] T 58fb54d17b9943cf8f21be2778663c72 P 2195f4b573b84fda85724f693eda2a36: stopping tablet replica
I20260810 04:09:11.892928 15634 raft_consensus.cc:2229] T 58fb54d17b9943cf8f21be2778663c72 P 2195f4b573b84fda85724f693eda2a36 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 04:09:11.893013 15634 raft_consensus.cc:2258] T 58fb54d17b9943cf8f21be2778663c72 P 2195f4b573b84fda85724f693eda2a36 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:09:11.893865 15633 tablet_replica.cc:330] T 58fb54d17b9943cf8f21be2778663c72 P cf99f2b9d85949f8ae4e5bbe8021050b: stopping tablet replica
I20260810 04:09:11.893949 15633 raft_consensus.cc:2229] T 58fb54d17b9943cf8f21be2778663c72 P cf99f2b9d85949f8ae4e5bbe8021050b [term 1 LEADER]: Raft consensus shutting down.
I20260810 04:09:11.894018 15633 raft_consensus.cc:2258] T 58fb54d17b9943cf8f21be2778663c72 P cf99f2b9d85949f8ae4e5bbe8021050b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:09:11.894670 15634 ts_tablet_manager.cc:1905] T 58fb54d17b9943cf8f21be2778663c72 P 2195f4b573b84fda85724f693eda2a36: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:09:11.894670 15633 ts_tablet_manager.cc:1905] T 58fb54d17b9943cf8f21be2778663c72 P cf99f2b9d85949f8ae4e5bbe8021050b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:09:11.896358 15633 ts_tablet_manager.cc:1918] T 58fb54d17b9943cf8f21be2778663c72 P cf99f2b9d85949f8ae4e5bbe8021050b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 04:09:11.896358 15634 ts_tablet_manager.cc:1918] T 58fb54d17b9943cf8f21be2778663c72 P 2195f4b573b84fda85724f693eda2a36: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 04:09:11.896436 15633 log.cc:1196] T 58fb54d17b9943cf8f21be2778663c72 P cf99f2b9d85949f8ae4e5bbe8021050b: Deleting WAL directory at /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/ts-1/wal/wals/58fb54d17b9943cf8f21be2778663c72
I20260810 04:09:11.896534 15634 log.cc:1196] T 58fb54d17b9943cf8f21be2778663c72 P 2195f4b573b84fda85724f693eda2a36: Deleting WAL directory at /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/ts-2/wal/wals/58fb54d17b9943cf8f21be2778663c72
I20260810 04:09:11.896741 15633 ts_tablet_manager.cc:1939] T 58fb54d17b9943cf8f21be2778663c72 P cf99f2b9d85949f8ae4e5bbe8021050b: Deleting consensus metadata
I20260810 04:09:11.896934 15634 ts_tablet_manager.cc:1939] T 58fb54d17b9943cf8f21be2778663c72 P 2195f4b573b84fda85724f693eda2a36: Deleting consensus metadata
I20260810 04:09:11.897156 15077 catalog_manager.cc:4841] TS cf99f2b9d85949f8ae4e5bbe8021050b (127.31.32.2:34627): tablet 58fb54d17b9943cf8f21be2778663c72 (table test-table [id=c5ec07266c0d4990b26f5d910c97be11]) successfully deleted
I20260810 04:09:11.897395 15080 catalog_manager.cc:4841] TS 2195f4b573b84fda85724f693eda2a36 (127.31.32.3:46777): tablet 58fb54d17b9943cf8f21be2778663c72 (table test-table [id=c5ec07266c0d4990b26f5d910c97be11]) successfully deleted
W20260810 04:09:11.904377 15080 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.32.1:39207 (error 108)
I20260810 04:09:11.904592 31872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskws89Io/build/release/bin/kudu with pid 15307
I20260810 04:09:11.907610 15609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:11.907430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.1:39207 (local address 127.0.0.1:44370)
0810 04:09:11.907529 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0810 04:09:11.907569 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.1:39207: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
W20260810 04:09:11.907723 15080 catalog_manager.cc:4584] TS df3980e9202d4bb586c7c6b817158342 (127.31.32.1:39207): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 58fb54d17b9943cf8f21be2778663c72: Network error: Client connection negotiation failed: client connection to 127.31.32.1:39207: connect: Connection refused (error 111)
I20260810 04:09:11.912573 31872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskws89Io/build/release/bin/kudu with pid 15457
I20260810 04:09:11.918844 31872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskws89Io/build/release/bin/kudu with pid 15059
I20260810 04:09:11.925832 31872 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskws89Io/build/release/bin/kudu
/tmp/dist-test-taskws89Io/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.32.62:44907
--webserver_interface=127.31.32.62
--webserver_port=44181
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.62.keytab
--principal=kudu/127.31.32.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.32.20:34365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.32.62:44907
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:09:12.004698 15636 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:09:12.004891 15636 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:09:12.004914 15636 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:09:12.006057 15636 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:09:12.006098 15636 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:09:12.006114 15636 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:09:12.006166 15636 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:09:12.006189 15636 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:09:12.007666 15636 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.32.20:34365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.32.62:44907
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.62.keytab
--principal=kudu/127.31.32.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.32.62:44907
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.32.62
--webserver_port=44181
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15636
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:09:12.007912 15636 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:09:12.008168 15636 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:09:12.012971 15643 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:09:12.012979 15642 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:09:12.012987 15645 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:09:12.013298 15636 server_base.cc:1034] running on GCE node
I20260810 04:09:12.013511 15636 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:09:12.013734 15636 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:09:12.014865 15636 hybrid_clock.cc:648] HybridClock initialized: now 1786334952014846 us; error 34 us; skew 500 ppm
Aug 10 04:09:12 dist-test-slave-8w9v krb5kdc[14461](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334952, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:09:12.017382 15636 init.cc:377] Logged in from keytab as kudu/127.31.32.62@KRBTEST.COM (short username kudu)
I20260810 04:09:12.017818 15636 webserver.cc:460] Webserver started at http://127.31.32.62:44181/ using document root <none> and password file <none>
I20260810 04:09:12.018044 15636 fs_manager.cc:359] Metadata directory not provided
I20260810 04:09:12.018124 15636 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:09:12.019822 15636 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 04:09:12.020550 15655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:09:12.020709 15636 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260810 04:09:12.020807 15636 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/master-0/data,/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/master-0/wal
uuid: "affc5cedaef749e2b7f7f15f09b0ecd0"
format_stamp: "Formatted at 2026-08-10 04:09:10 on dist-test-slave-8w9v"
server_key: "c5e671ecb9c5e9286496d0ca6022cf05"
server_key_iv: "e2e14dd5d26b733c9145fbb549cb1f30"
server_key_version: "encryptionkey@0"
I20260810 04:09:12.021140 15636 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:09:12.045217 15636 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:09:12.047587 15636 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:09:12.047739 15636 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:09:12.052925 15636 rpc_server.cc:307] RPC server started. Bound to: 127.31.32.62:44907
I20260810 04:09:12.052959 15710 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.32.62:44907 every 8 connection(s)
I20260810 04:09:12.053337 15636 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/master-0/data/info.pb
I20260810 04:09:12.054926 15711 sys_catalog.cc:263] Verifying existing consensus state
I20260810 04:09:12.055503 15711 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P affc5cedaef749e2b7f7f15f09b0ecd0: Bootstrap starting.
I20260810 04:09:12.060806 31872 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskws89Io/build/release/bin/kudu as pid 15636
I20260810 04:09:12.061156 31872 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskws89Io/build/release/bin/kudu
/tmp/dist-test-taskws89Io/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.32.1:39207
--local_ip_for_outbound_sockets=127.31.32.1
--tserver_master_addrs=127.31.32.62:44907
--webserver_port=45699
--webserver_interface=127.31.32.1
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.1.keytab
--principal=kudu/127.31.32.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.32.20:34365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 04:09:12.064389 15711 log.cc:824] T 00000000000000000000000000000000 P affc5cedaef749e2b7f7f15f09b0ecd0: Log is configured to *not* fsync() on all Append() calls
I20260810 04:09:12.066480 15711 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P affc5cedaef749e2b7f7f15f09b0ecd0: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 04:09:12.066766 15711 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P affc5cedaef749e2b7f7f15f09b0ecd0: Bootstrap complete.
I20260810 04:09:12.068989 15711 raft_consensus.cc:348] T 00000000000000000000000000000000 P affc5cedaef749e2b7f7f15f09b0ecd0 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "affc5cedaef749e2b7f7f15f09b0ecd0" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 44907 } }
I20260810 04:09:12.069185 15711 raft_consensus.cc:729] T 00000000000000000000000000000000 P affc5cedaef749e2b7f7f15f09b0ecd0 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: affc5cedaef749e2b7f7f15f09b0ecd0, State: Initialized, Role: FOLLOWER
I20260810 04:09:12.069362 15711 consensus_queue.cc:260] T 00000000000000000000000000000000 P affc5cedaef749e2b7f7f15f09b0ecd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "affc5cedaef749e2b7f7f15f09b0ecd0" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 44907 } }
I20260810 04:09:12.069423 15711 raft_consensus.cc:388] T 00000000000000000000000000000000 P affc5cedaef749e2b7f7f15f09b0ecd0 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 04:09:12.069478 15711 raft_consensus.cc:482] T 00000000000000000000000000000000 P affc5cedaef749e2b7f7f15f09b0ecd0 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 04:09:12.069523 15711 raft_consensus.cc:3037] T 00000000000000000000000000000000 P affc5cedaef749e2b7f7f15f09b0ecd0 [term 2 FOLLOWER]: Advancing to term 3
I20260810 04:09:12.070494 15711 raft_consensus.cc:504] T 00000000000000000000000000000000 P affc5cedaef749e2b7f7f15f09b0ecd0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "affc5cedaef749e2b7f7f15f09b0ecd0" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 44907 } }
I20260810 04:09:12.070580 15711 leader_election.cc:302] T 00000000000000000000000000000000 P affc5cedaef749e2b7f7f15f09b0ecd0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: affc5cedaef749e2b7f7f15f09b0ecd0; no voters:
I20260810 04:09:12.070756 15711 leader_election.cc:288] T 00000000000000000000000000000000 P affc5cedaef749e2b7f7f15f09b0ecd0 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260810 04:09:12.070825 15716 raft_consensus.cc:2781] T 00000000000000000000000000000000 P affc5cedaef749e2b7f7f15f09b0ecd0 [term 3 FOLLOWER]: Leader election won for term 3
I20260810 04:09:12.070990 15716 raft_consensus.cc:686] T 00000000000000000000000000000000 P affc5cedaef749e2b7f7f15f09b0ecd0 [term 3 LEADER]: Becoming Leader. State: Replica: affc5cedaef749e2b7f7f15f09b0ecd0, State: Running, Role: LEADER
I20260810 04:09:12.071038 15711 sys_catalog.cc:564] T 00000000000000000000000000000000 P affc5cedaef749e2b7f7f15f09b0ecd0 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:09:12.071107 15716 consensus_queue.cc:237] T 00000000000000000000000000000000 P affc5cedaef749e2b7f7f15f09b0ecd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "affc5cedaef749e2b7f7f15f09b0ecd0" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 44907 } }
I20260810 04:09:12.071398 15717 sys_catalog.cc:455] T 00000000000000000000000000000000 P affc5cedaef749e2b7f7f15f09b0ecd0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "affc5cedaef749e2b7f7f15f09b0ecd0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "affc5cedaef749e2b7f7f15f09b0ecd0" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 44907 } } }
I20260810 04:09:12.071487 15717 sys_catalog.cc:458] T 00000000000000000000000000000000 P affc5cedaef749e2b7f7f15f09b0ecd0 [sys.catalog]: This master's current role is: LEADER
I20260810 04:09:12.071731 15718 sys_catalog.cc:455] T 00000000000000000000000000000000 P affc5cedaef749e2b7f7f15f09b0ecd0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader affc5cedaef749e2b7f7f15f09b0ecd0. Latest consensus state: current_term: 3 leader_uuid: "affc5cedaef749e2b7f7f15f09b0ecd0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "affc5cedaef749e2b7f7f15f09b0ecd0" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 44907 } } }
I20260810 04:09:12.071866 15718 sys_catalog.cc:458] T 00000000000000000000000000000000 P affc5cedaef749e2b7f7f15f09b0ecd0 [sys.catalog]: This master's current role is: LEADER
I20260810 04:09:12.071897 15724 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 04:09:12.072671 15724 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 04:09:12.072769 15724 catalog_manager.cc:1259] Loaded cluster ID: 3cb9a05ff41b4b4f9928b6cfef7d4d8e
I20260810 04:09:12.072983 15724 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260810 04:09:12.073342 15724 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f74fb7f69ad google::LogMessage::Fail() at ??:0
@ 0x7f74fb7f988a google::LogMessage::SendToLog() at ??:0
@ 0x7f74fb7f64de google::LogMessage::Flush() at ??:0
@ 0x7f74fb7f7a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f750371b0f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f75037248b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f74fd6c40d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f74fd6b9954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f74fe51d6db start_thread at ??:0
@ 0x7f74fa2f461f clone at ??:0
W20260810 04:09:12.161624 15714 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:09:12.161849 15714 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:09:12.161875 15714 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:09:12.163039 15714 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:09:12.163100 15714 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:09:12.163127 15714 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.32.1
I20260810 04:09:12.164942 15714 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.32.20:34365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.1.keytab
--principal=kudu/127.31.32.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.32.1:39207
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.32.1
--webserver_port=45699
--tserver_master_addrs=127.31.32.62:44907
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15714
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.32.1
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:09:12.165172 15714 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:09:12.165427 15714 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:09:12.170624 15751 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:09:12.170706 15754 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:09:12.170653 15750 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:09:12.171288 15714 server_base.cc:1034] running on GCE node
I20260810 04:09:12.171478 15714 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:09:12.171738 15714 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:09:12.172910 15714 hybrid_clock.cc:648] HybridClock initialized: now 1786334952172897 us; error 27 us; skew 500 ppm
Aug 10 04:09:12 dist-test-slave-8w9v krb5kdc[14461](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334952, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:09:12.175498 15714 init.cc:377] Logged in from keytab as kudu/127.31.32.1@KRBTEST.COM (short username kudu)
I20260810 04:09:12.176023 15714 webserver.cc:460] Webserver started at http://127.31.32.1:45699/ using document root <none> and password file <none>
I20260810 04:09:12.176293 15714 fs_manager.cc:359] Metadata directory not provided
I20260810 04:09:12.176339 15714 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:09:12.178099 15714 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 04:09:12.178851 15761 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:09:12.179046 15714 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:09:12.179159 15714 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/ts-0/data,/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/ts-0/wal
uuid: "df3980e9202d4bb586c7c6b817158342"
format_stamp: "Formatted at 2026-08-10 04:09:10 on dist-test-slave-8w9v"
server_key: "b92f55ed48a4450688ecb93317a5422c"
server_key_iv: "f96d477058c33bdca2dbf94b4555b30f"
server_key_version: "encryptionkey@0"
I20260810 04:09:12.179530 15714 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:09:12.194167 15714 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:09:12.196518 15714 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:09:12.196664 15714 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:09:12.196933 15714 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:09:12.197419 15769 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260810 04:09:12.198827 15714 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260810 04:09:12.198882 15714 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260810 04:09:12.198905 15714 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260810 04:09:12.199560 15714 ts_tablet_manager.cc:610] Registered 1 tablets
I20260810 04:09:12.199606 15714 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260810 04:09:12.199651 15769 tablet_bootstrap.cc:492] T 58fb54d17b9943cf8f21be2778663c72 P df3980e9202d4bb586c7c6b817158342: Bootstrap starting.
I20260810 04:09:12.208583 15714 rpc_server.cc:307] RPC server started. Bound to: 127.31.32.1:39207
I20260810 04:09:12.208807 15886 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.32.1:39207 every 8 connection(s)
I20260810 04:09:12.209086 15714 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/ts-0/data/info.pb
I20260810 04:09:12.212339 15769 log.cc:824] T 58fb54d17b9943cf8f21be2778663c72 P df3980e9202d4bb586c7c6b817158342: Log is configured to *not* fsync() on all Append() calls
I20260810 04:09:12.214356 15769 tablet_bootstrap.cc:492] T 58fb54d17b9943cf8f21be2778663c72 P df3980e9202d4bb586c7c6b817158342: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 04:09:12.214643 15769 tablet_bootstrap.cc:492] T 58fb54d17b9943cf8f21be2778663c72 P df3980e9202d4bb586c7c6b817158342: Bootstrap complete.
I20260810 04:09:12.215592 15769 ts_tablet_manager.cc:1397] T 58fb54d17b9943cf8f21be2778663c72 P df3980e9202d4bb586c7c6b817158342: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.007s
I20260810 04:09:12.216675 15769 raft_consensus.cc:348] T 58fb54d17b9943cf8f21be2778663c72 P df3980e9202d4bb586c7c6b817158342 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2195f4b573b84fda85724f693eda2a36" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 46777 } } peers { permanent_uuid: "cf99f2b9d85949f8ae4e5bbe8021050b" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 34627 } } peers { permanent_uuid: "df3980e9202d4bb586c7c6b817158342" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 39207 } }
I20260810 04:09:12.216898 15769 raft_consensus.cc:729] T 58fb54d17b9943cf8f21be2778663c72 P df3980e9202d4bb586c7c6b817158342 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: df3980e9202d4bb586c7c6b817158342, State: Initialized, Role: FOLLOWER
I20260810 04:09:12.217031 15769 consensus_queue.cc:260] T 58fb54d17b9943cf8f21be2778663c72 P df3980e9202d4bb586c7c6b817158342 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2195f4b573b84fda85724f693eda2a36" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 46777 } } peers { permanent_uuid: "cf99f2b9d85949f8ae4e5bbe8021050b" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 34627 } } peers { permanent_uuid: "df3980e9202d4bb586c7c6b817158342" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 39207 } }
I20260810 04:09:12.217348 15769 ts_tablet_manager.cc:1428] T 58fb54d17b9943cf8f21be2778663c72 P df3980e9202d4bb586c7c6b817158342: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20260810 04:09:12.217444 31872 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskws89Io/build/release/bin/kudu as pid 15714
I20260810 04:09:12.217724 31872 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskws89Io/build/release/bin/kudu
/tmp/dist-test-taskws89Io/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.32.2:34627
--local_ip_for_outbound_sockets=127.31.32.2
--tserver_master_addrs=127.31.32.62:44907
--webserver_port=42345
--webserver_interface=127.31.32.2
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.2.keytab
--principal=kudu/127.31.32.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.32.20:34365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:09:12.258332 15889 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:09:12.209827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:44907 (local address 127.31.32.1:59017)
0810 04:09:12.210000 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0810 04:09:12.210045 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 04:09:12.210763 (+ 718us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:09:12.258130 (+ 47367us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.62:44907: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":100,"threads_started":1}
W20260810 04:09:12.258621 15887 heartbeater.cc:644] Failed to heartbeat to 127.31.32.62:44907 (0 consecutive failures): Network error: Failed to ping master at 127.31.32.62:44907: Client connection negotiation failed: client connection to 127.31.32.62:44907: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260810 04:09:12.259024 15889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:12.258899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:44907 (local address 127.31.32.1:37795)
0810 04:09:12.258956 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0810 04:09:12.259000 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.62:44907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260810 04:09:12.259389 15889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:12.259297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:44907 (local address 127.31.32.1:42257)
0810 04:09:12.259342 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0810 04:09:12.259369 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.62:44907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260810 04:09:12.259506 15887 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 04:09:12.304502 15892 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:09:12.304699 15892 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:09:12.304720 15892 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:09:12.305912 15892 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:09:12.305954 15892 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:09:12.305980 15892 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.32.2
I20260810 04:09:12.307382 15892 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.32.20:34365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.2.keytab
--principal=kudu/127.31.32.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.32.2:34627
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.32.2
--webserver_port=42345
--tserver_master_addrs=127.31.32.62:44907
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15892
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.32.2
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:09:12.307611 15892 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:09:12.307829 15892 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:09:12.312592 15897 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:09:12.312687 15898 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:09:12.312736 15892 server_base.cc:1034] running on GCE node
W20260810 04:09:12.312604 15900 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:09:12.313086 15892 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:09:12.313344 15892 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:09:12.314503 15892 hybrid_clock.cc:648] HybridClock initialized: now 1786334952314499 us; error 35 us; skew 500 ppm
Aug 10 04:09:12 dist-test-slave-8w9v krb5kdc[14461](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334952, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:09:12.317016 15892 init.cc:377] Logged in from keytab as kudu/127.31.32.2@KRBTEST.COM (short username kudu)
I20260810 04:09:12.317446 15892 webserver.cc:460] Webserver started at http://127.31.32.2:42345/ using document root <none> and password file <none>
I20260810 04:09:12.317632 15892 fs_manager.cc:359] Metadata directory not provided
I20260810 04:09:12.317670 15892 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:09:12.318876 15892 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:09:12.319399 15907 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:09:12.319610 15892 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:09:12.319684 15892 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/ts-1/data,/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/ts-1/wal
uuid: "cf99f2b9d85949f8ae4e5bbe8021050b"
format_stamp: "Formatted at 2026-08-10 04:09:11 on dist-test-slave-8w9v"
server_key: "80e271ba77cc9983241470c2b1abf7a9"
server_key_iv: "945fbf70d98a4ebfbc2c8275c88b2169"
server_key_version: "encryptionkey@0"
I20260810 04:09:12.319993 15892 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:09:12.339931 15892 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:09:12.342211 15892 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:09:12.342325 15892 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:09:12.342504 15892 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:09:12.342851 15892 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:09:12.342888 15892 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:09:12.342933 15892 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:09:12.342959 15892 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:09:12.349246 15892 rpc_server.cc:307] RPC server started. Bound to: 127.31.32.2:34627
I20260810 04:09:12.349313 16029 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.32.2:34627 every 8 connection(s)
I20260810 04:09:12.349633 15892 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/ts-1/data/info.pb
I20260810 04:09:12.350636 16032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:12.350259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:44907 (local address 127.31.32.2:50577)
0810 04:09:12.350427 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0810 04:09:12.350500 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.62:44907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":93,"threads_started":1}
W20260810 04:09:12.351061 16030 heartbeater.cc:644] Failed to heartbeat to 127.31.32.62:44907 (0 consecutive failures): Network error: Failed to ping master at 127.31.32.62:44907: Client connection negotiation failed: client connection to 127.31.32.62:44907: connect: Connection refused (error 111)
I20260810 04:09:12.351369 16032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:12.351262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:44907 (local address 127.31.32.2:56887)
0810 04:09:12.351305 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0810 04:09:12.351350 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.62:44907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260810 04:09:12.351652 16032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:12.351580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:44907 (local address 127.31.32.2:59863)
0810 04:09:12.351619 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0810 04:09:12.351639 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.62:44907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260810 04:09:12.351773 16030 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260810 04:09:12.353615 31872 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskws89Io/build/release/bin/kudu as pid 15892
I20260810 04:09:12.353853 31872 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskws89Io/build/release/bin/kudu
/tmp/dist-test-taskws89Io/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.32.3:46777
--local_ip_for_outbound_sockets=127.31.32.3
--tserver_master_addrs=127.31.32.62:44907
--webserver_port=33751
--webserver_interface=127.31.32.3
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.3.keytab
--principal=kudu/127.31.32.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.32.20:34365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:09:12.434424 16033 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:09:12.434619 16033 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:09:12.434639 16033 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:09:12.435760 16033 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:09:12.435802 16033 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:09:12.435827 16033 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.32.3
I20260810 04:09:12.437295 16033 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.32.20:34365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.3.keytab
--principal=kudu/127.31.32.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.32.3:46777
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.32.3
--webserver_port=33751
--tserver_master_addrs=127.31.32.62:44907
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16033
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.32.3
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:09:12.437551 16033 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:09:12.437763 16033 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:09:12.442562 16039 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:09:12.442566 16042 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:09:12.442631 16033 server_base.cc:1034] running on GCE node
W20260810 04:09:12.442598 16040 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:09:12.443042 16033 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:09:12.443252 16033 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:09:12.444391 16033 hybrid_clock.cc:648] HybridClock initialized: now 1786334952444380 us; error 30 us; skew 500 ppm
Aug 10 04:09:12 dist-test-slave-8w9v krb5kdc[14461](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334952, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:09:12.446842 16033 init.cc:377] Logged in from keytab as kudu/127.31.32.3@KRBTEST.COM (short username kudu)
I20260810 04:09:12.447216 16033 webserver.cc:460] Webserver started at http://127.31.32.3:33751/ using document root <none> and password file <none>
I20260810 04:09:12.447408 16033 fs_manager.cc:359] Metadata directory not provided
I20260810 04:09:12.447449 16033 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:09:12.448758 16033 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260810 04:09:12.449292 16049 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:09:12.449456 16033 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 04:09:12.449559 16033 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/ts-2/data,/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/ts-2/wal
uuid: "2195f4b573b84fda85724f693eda2a36"
format_stamp: "Formatted at 2026-08-10 04:09:11 on dist-test-slave-8w9v"
server_key: "70cd799f0634144a7b3b59ed05934c23"
server_key_iv: "3673ef5a08d4dc00e0b0e6746192deda"
server_key_version: "encryptionkey@0"
I20260810 04:09:12.449868 16033 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:09:12.459081 16033 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:09:12.461387 16033 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:09:12.461584 16033 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:09:12.461828 16033 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:09:12.462174 16033 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:09:12.462208 16033 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:09:12.462231 16033 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:09:12.462275 16033 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:09:12.467983 16033 rpc_server.cc:307] RPC server started. Bound to: 127.31.32.3:46777
I20260810 04:09:12.468075 16180 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.32.3:46777 every 8 connection(s)
I20260810 04:09:12.468437 16033 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786334794676589-31872-0/minicluster-data/ts-2/data/info.pb
I20260810 04:09:12.469130 31872 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskws89Io/build/release/bin/kudu as pid 16033
I20260810 04:09:12.469570 16183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:12.469113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:44907 (local address 127.31.32.3:35627)
0810 04:09:12.469340 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0810 04:09:12.469437 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.62:44907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":126,"threads_started":1}
W20260810 04:09:12.469990 16181 heartbeater.cc:644] Failed to heartbeat to 127.31.32.62:44907 (0 consecutive failures): Network error: Failed to ping master at 127.31.32.62:44907: Client connection negotiation failed: client connection to 127.31.32.62:44907: connect: Connection refused (error 111)
I20260810 04:09:12.470270 16183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:12.470179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:44907 (local address 127.31.32.3:46135)
0810 04:09:12.470225 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0810 04:09:12.470251 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.62:44907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260810 04:09:12.470592 16183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:12.470518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:44907 (local address 127.31.32.3:47941)
0810 04:09:12.470556 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0810 04:09:12.470577 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.62:44907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260810 04:09:12.470709 16181 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260810 04:09:13.260095 16185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:13.259863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:44907 (local address 127.31.32.1:46897)
0810 04:09:13.259989 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0810 04:09:13.260026 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.62:44907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":64,"threads_started":1}
I20260810 04:09:13.352286 16186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:13.352050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:44907 (local address 127.31.32.2:51333)
0810 04:09:13.352199 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0810 04:09:13.352232 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.62:44907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":60,"threads_started":1}
I20260810 04:09:13.471221 16187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:13.471027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:44907 (local address 127.31.32.3:35975)
0810 04:09:13.471143 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0810 04:09:13.471176 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.62:44907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":61,"threads_started":1}
I20260810 04:09:13.531458 16188 raft_consensus.cc:482] T 58fb54d17b9943cf8f21be2778663c72 P df3980e9202d4bb586c7c6b817158342 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 04:09:13.531627 16188 raft_consensus.cc:504] T 58fb54d17b9943cf8f21be2778663c72 P df3980e9202d4bb586c7c6b817158342 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2195f4b573b84fda85724f693eda2a36" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 46777 } } peers { permanent_uuid: "cf99f2b9d85949f8ae4e5bbe8021050b" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 34627 } } peers { permanent_uuid: "df3980e9202d4bb586c7c6b817158342" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 39207 } }
I20260810 04:09:13.532007 16188 leader_election.cc:288] T 58fb54d17b9943cf8f21be2778663c72 P df3980e9202d4bb586c7c6b817158342 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2195f4b573b84fda85724f693eda2a36 (127.31.32.3:46777), cf99f2b9d85949f8ae4e5bbe8021050b (127.31.32.2:34627)
Aug 10 04:09:13 dist-test-slave-8w9v krb5kdc[14461](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334952, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.1@KRBTEST.COM for kudu/127.31.32.2@KRBTEST.COM
Aug 10 04:09:13 dist-test-slave-8w9v krb5kdc[14461](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334952, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.1@KRBTEST.COM for kudu/127.31.32.3@KRBTEST.COM
I20260810 04:09:13.542222 16191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:13.532347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.1:36055 (local address 127.31.32.2:34627)
0810 04:09:13.532485 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0810 04:09:13.532493 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 04:09:13.532520 (+ 27us) server_negotiation.cc:407] Connection header received
0810 04:09:13.533101 (+ 581us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:09:13.533103 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:09:13.533163 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:09:13.533316 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:09:13.534039 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:09:13.534949 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:09:13.535917 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:09:13.536389 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:09:13.536509 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:09:13.538743 (+ 2234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:09:13.538746 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:09:13.538746 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:09:13.538755 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:09:13.540614 (+ 1859us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:09:13.541000 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:09:13.541003 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:09:13.541003 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:09:13.541109 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:09:13.541367 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:09:13.541368 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:09:13.541368 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:09:13.541589 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:09:13.541672 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0810 04:09:13.541961 (+ 289us) server_negotiation.cc:299] Negotiation successful
0810 04:09:13.542102 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":64,"threads_started":1}
I20260810 04:09:13.542212 16189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:13.532172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.2:34627 (local address 127.31.32.1:36055)
0810 04:09:13.532296 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0810 04:09:13.532306 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 04:09:13.532370 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:09:13.533441 (+ 1071us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:09:13.533442 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:09:13.533693 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:09:13.533897 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:09:13.533903 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:09:13.535045 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:09:13.535045 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:09:13.535816 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:09:13.535818 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:09:13.536482 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:09:13.536483 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:09:13.536599 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:09:13.537209 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:09:13.537213 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:09:13.538646 (+ 1433us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:09:13.540687 (+ 2041us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:09:13.540693 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:09:13.540695 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:09:13.540897 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:09:13.541180 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:09:13.541181 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:09:13.541182 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:09:13.541285 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:09:13.541667 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:09:13.541673 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:09:13.541788 (+ 115us) client_negotiation.cc:769] Sending connection context
0810 04:09:13.541891 (+ 103us) client_negotiation.cc:240] Negotiation successful
0810 04:09:13.542096 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"mutex_wait_us":41,"thread_start_us":50,"threads_started":1}
I20260810 04:09:13.542265 16190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:13.532273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.1:54427 (local address 127.31.32.3:46777)
0810 04:09:13.532444 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0810 04:09:13.532458 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 04:09:13.532494 (+ 36us) server_negotiation.cc:407] Connection header received
0810 04:09:13.533100 (+ 606us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:09:13.533103 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:09:13.533162 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:09:13.533318 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:09:13.534017 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:09:13.534939 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:09:13.535909 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:09:13.536381 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:09:13.536509 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:09:13.539108 (+ 2599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:09:13.539111 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:09:13.539111 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:09:13.539120 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:09:13.540818 (+ 1698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:09:13.541157 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:09:13.541159 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:09:13.541160 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:09:13.541248 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:09:13.541451 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:09:13.541452 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:09:13.541452 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:09:13.541669 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:09:13.541756 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0810 04:09:13.541955 (+ 199us) server_negotiation.cc:299] Negotiation successful
0810 04:09:13.542148 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":73,"threads_started":1}
I20260810 04:09:13.542311 16185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:13.532160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.3:46777 (local address 127.31.32.1:54427)
0810 04:09:13.532217 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0810 04:09:13.532232 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 04:09:13.532323 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:09:13.533368 (+ 1045us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:09:13.533370 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:09:13.533704 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:09:13.533897 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:09:13.533903 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:09:13.535019 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:09:13.535021 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:09:13.535818 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:09:13.535820 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:09:13.536485 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:09:13.536486 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:09:13.536599 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:09:13.537164 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:09:13.537173 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:09:13.539002 (+ 1829us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:09:13.540891 (+ 1889us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:09:13.540892 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:09:13.540893 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:09:13.541049 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:09:13.541323 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:09:13.541324 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:09:13.541324 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:09:13.541389 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:09:13.541740 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:09:13.541741 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:09:13.541797 (+ 56us) client_negotiation.cc:769] Sending connection context
0810 04:09:13.541891 (+ 94us) client_negotiation.cc:240] Negotiation successful
0810 04:09:13.542169 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39,"mutex_wait_us":177}
W20260810 04:09:13.543349 15762 leader_election.cc:341] T 58fb54d17b9943cf8f21be2778663c72 P df3980e9202d4bb586c7c6b817158342 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cf99f2b9d85949f8ae4e5bbe8021050b (127.31.32.2:34627): Not found: Tablet not found: 58fb54d17b9943cf8f21be2778663c72
W20260810 04:09:13.543430 15765 leader_election.cc:341] T 58fb54d17b9943cf8f21be2778663c72 P df3980e9202d4bb586c7c6b817158342 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2195f4b573b84fda85724f693eda2a36 (127.31.32.3:46777): Not found: Tablet not found: 58fb54d17b9943cf8f21be2778663c72
I20260810 04:09:13.543526 15765 leader_election.cc:302] T 58fb54d17b9943cf8f21be2778663c72 P df3980e9202d4bb586c7c6b817158342 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3980e9202d4bb586c7c6b817158342; no voters: 2195f4b573b84fda85724f693eda2a36, cf99f2b9d85949f8ae4e5bbe8021050b
I20260810 04:09:13.543692 16188 raft_consensus.cc:2726] T 58fb54d17b9943cf8f21be2778663c72 P df3980e9202d4bb586c7c6b817158342 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 04:09:14.260867 16192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:14.260635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:44907 (local address 127.31.32.1:60113)
0810 04:09:14.260772 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0810 04:09:14.260813 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.62:44907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":63,"threads_started":1}
I20260810 04:09:14.352876 16193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:14.352676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:44907 (local address 127.31.32.2:58871)
0810 04:09:14.352802 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0810 04:09:14.352840 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.62:44907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":62,"threads_started":1}
I20260810 04:09:14.471962 16194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:14.471730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:44907 (local address 127.31.32.3:36477)
0810 04:09:14.471879 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0810 04:09:14.471919 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.62:44907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":68,"threads_started":1}
I20260810 04:09:15.053699 16195 raft_consensus.cc:482] T 58fb54d17b9943cf8f21be2778663c72 P df3980e9202d4bb586c7c6b817158342 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 04:09:15.053798 16195 raft_consensus.cc:504] T 58fb54d17b9943cf8f21be2778663c72 P df3980e9202d4bb586c7c6b817158342 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2195f4b573b84fda85724f693eda2a36" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 46777 } } peers { permanent_uuid: "cf99f2b9d85949f8ae4e5bbe8021050b" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 34627 } } peers { permanent_uuid: "df3980e9202d4bb586c7c6b817158342" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 39207 } }
I20260810 04:09:15.054016 16195 leader_election.cc:288] T 58fb54d17b9943cf8f21be2778663c72 P df3980e9202d4bb586c7c6b817158342 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2195f4b573b84fda85724f693eda2a36 (127.31.32.3:46777), cf99f2b9d85949f8ae4e5bbe8021050b (127.31.32.2:34627)
W20260810 04:09:15.054488 15765 leader_election.cc:341] T 58fb54d17b9943cf8f21be2778663c72 P df3980e9202d4bb586c7c6b817158342 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2195f4b573b84fda85724f693eda2a36 (127.31.32.3:46777): Not found: Tablet not found: 58fb54d17b9943cf8f21be2778663c72
W20260810 04:09:15.054602 15762 leader_election.cc:341] T 58fb54d17b9943cf8f21be2778663c72 P df3980e9202d4bb586c7c6b817158342 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cf99f2b9d85949f8ae4e5bbe8021050b (127.31.32.2:34627): Not found: Tablet not found: 58fb54d17b9943cf8f21be2778663c72
I20260810 04:09:15.054636 15765 leader_election.cc:302] T 58fb54d17b9943cf8f21be2778663c72 P df3980e9202d4bb586c7c6b817158342 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3980e9202d4bb586c7c6b817158342; no voters: 2195f4b573b84fda85724f693eda2a36, cf99f2b9d85949f8ae4e5bbe8021050b
I20260810 04:09:15.054714 16195 raft_consensus.cc:2726] T 58fb54d17b9943cf8f21be2778663c72 P df3980e9202d4bb586c7c6b817158342 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 04:09:15.261515 16196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:15.261278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:44907 (local address 127.31.32.1:46467)
0810 04:09:15.261435 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0810 04:09:15.261468 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.62:44907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":100,"threads_started":1}
I20260810 04:09:15.353475 16197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:15.353269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:44907 (local address 127.31.32.2:56729)
0810 04:09:15.353405 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0810 04:09:15.353437 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.62:44907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":76,"threads_started":1}
I20260810 04:09:15.472640 16198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:15.472443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:44907 (local address 127.31.32.3:52999)
0810 04:09:15.472564 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0810 04:09:15.472600 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.62:44907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":65,"threads_started":1}
I20260810 04:09:16.262215 16199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:16.261971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:44907 (local address 127.31.32.1:41675)
0810 04:09:16.262127 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0810 04:09:16.262167 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.62:44907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":69,"threads_started":1}
I20260810 04:09:16.354146 16200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:16.353914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:44907 (local address 127.31.32.2:40731)
0810 04:09:16.354050 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0810 04:09:16.354088 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.62:44907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":62,"threads_started":1}
I20260810 04:09:16.473239 16201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:16.473055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:44907 (local address 127.31.32.3:58661)
0810 04:09:16.473169 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0810 04:09:16.473203 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.62:44907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":65,"threads_started":1}
I20260810 04:09:17.262995 16202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:17.262743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:44907 (local address 127.31.32.1:41295)
0810 04:09:17.262914 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0810 04:09:17.262945 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.62:44907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":110,"threads_started":1}
I20260810 04:09:17.354831 16203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:17.354605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:44907 (local address 127.31.32.2:41471)
0810 04:09:17.354752 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0810 04:09:17.354790 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.62:44907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":75,"threads_started":1}
I20260810 04:09:17.473922 16204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:17.473685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:44907 (local address 127.31.32.3:47043)
0810 04:09:17.473849 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0810 04:09:17.473882 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.62:44907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":101,"threads_started":1}
I20260810 04:09:17.689811 16205 raft_consensus.cc:482] T 58fb54d17b9943cf8f21be2778663c72 P df3980e9202d4bb586c7c6b817158342 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 04:09:17.689917 16205 raft_consensus.cc:504] T 58fb54d17b9943cf8f21be2778663c72 P df3980e9202d4bb586c7c6b817158342 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2195f4b573b84fda85724f693eda2a36" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 46777 } } peers { permanent_uuid: "cf99f2b9d85949f8ae4e5bbe8021050b" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 34627 } } peers { permanent_uuid: "df3980e9202d4bb586c7c6b817158342" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 39207 } }
I20260810 04:09:17.690122 16205 leader_election.cc:288] T 58fb54d17b9943cf8f21be2778663c72 P df3980e9202d4bb586c7c6b817158342 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2195f4b573b84fda85724f693eda2a36 (127.31.32.3:46777), cf99f2b9d85949f8ae4e5bbe8021050b (127.31.32.2:34627)
W20260810 04:09:17.690584 15765 leader_election.cc:341] T 58fb54d17b9943cf8f21be2778663c72 P df3980e9202d4bb586c7c6b817158342 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2195f4b573b84fda85724f693eda2a36 (127.31.32.3:46777): Not found: Tablet not found: 58fb54d17b9943cf8f21be2778663c72
W20260810 04:09:17.690667 15762 leader_election.cc:341] T 58fb54d17b9943cf8f21be2778663c72 P df3980e9202d4bb586c7c6b817158342 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cf99f2b9d85949f8ae4e5bbe8021050b (127.31.32.2:34627): Not found: Tablet not found: 58fb54d17b9943cf8f21be2778663c72
I20260810 04:09:17.690750 15762 leader_election.cc:302] T 58fb54d17b9943cf8f21be2778663c72 P df3980e9202d4bb586c7c6b817158342 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3980e9202d4bb586c7c6b817158342; no voters: 2195f4b573b84fda85724f693eda2a36, cf99f2b9d85949f8ae4e5bbe8021050b
I20260810 04:09:17.690838 16205 raft_consensus.cc:2726] T 58fb54d17b9943cf8f21be2778663c72 P df3980e9202d4bb586c7c6b817158342 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 04:09:18.263724 16206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:18.263499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:44907 (local address 127.31.32.1:38039)
0810 04:09:18.263636 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0810 04:09:18.263674 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.62:44907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":70,"threads_started":1}
I20260810 04:09:18.355525 16207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:18.355324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:44907 (local address 127.31.32.2:42045)
0810 04:09:18.355448 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0810 04:09:18.355483 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.62:44907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":63,"threads_started":1}
I20260810 04:09:18.474601 16208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:18.474384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:44907 (local address 127.31.32.3:41837)
0810 04:09:18.474529 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0810 04:09:18.474562 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.62:44907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":78,"threads_started":1}
I20260810 04:09:19.264544 16209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:19.264330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:44907 (local address 127.31.32.1:47471)
0810 04:09:19.264459 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0810 04:09:19.264496 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.62:44907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":68,"threads_started":1}
I20260810 04:09:19.356302 16210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:19.355966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:44907 (local address 127.31.32.2:54357)
0810 04:09:19.356213 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0810 04:09:19.356258 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.62:44907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":87,"threads_started":1}
I20260810 04:09:19.475238 16211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:19.475038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:44907 (local address 127.31.32.3:46527)
0810 04:09:19.475162 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0810 04:09:19.475197 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.62:44907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":65,"threads_started":1}
I20260810 04:09:20.265252 16212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:20.265040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:44907 (local address 127.31.32.1:35461)
0810 04:09:20.265167 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0810 04:09:20.265204 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.62:44907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":64,"threads_started":1}
I20260810 04:09:20.356969 16213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:20.356745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:44907 (local address 127.31.32.2:56973)
0810 04:09:20.356893 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0810 04:09:20.356928 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.62:44907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":76,"threads_started":1}
I20260810 04:09:20.475914 16214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:20.475664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:44907 (local address 127.31.32.3:34085)
0810 04:09:20.475823 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0810 04:09:20.475868 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.62:44907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":65,"threads_started":1}
I20260810 04:09:20.746052 16215 raft_consensus.cc:482] T 58fb54d17b9943cf8f21be2778663c72 P df3980e9202d4bb586c7c6b817158342 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 04:09:20.746153 16215 raft_consensus.cc:504] T 58fb54d17b9943cf8f21be2778663c72 P df3980e9202d4bb586c7c6b817158342 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2195f4b573b84fda85724f693eda2a36" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 46777 } } peers { permanent_uuid: "cf99f2b9d85949f8ae4e5bbe8021050b" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 34627 } } peers { permanent_uuid: "df3980e9202d4bb586c7c6b817158342" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 39207 } }
I20260810 04:09:20.746321 16215 leader_election.cc:288] T 58fb54d17b9943cf8f21be2778663c72 P df3980e9202d4bb586c7c6b817158342 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2195f4b573b84fda85724f693eda2a36 (127.31.32.3:46777), cf99f2b9d85949f8ae4e5bbe8021050b (127.31.32.2:34627)
W20260810 04:09:20.746758 15765 leader_election.cc:341] T 58fb54d17b9943cf8f21be2778663c72 P df3980e9202d4bb586c7c6b817158342 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2195f4b573b84fda85724f693eda2a36 (127.31.32.3:46777): Not found: Tablet not found: 58fb54d17b9943cf8f21be2778663c72
W20260810 04:09:20.746869 15762 leader_election.cc:341] T 58fb54d17b9943cf8f21be2778663c72 P df3980e9202d4bb586c7c6b817158342 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cf99f2b9d85949f8ae4e5bbe8021050b (127.31.32.2:34627): Not found: Tablet not found: 58fb54d17b9943cf8f21be2778663c72
I20260810 04:09:20.746960 15762 leader_election.cc:302] T 58fb54d17b9943cf8f21be2778663c72 P df3980e9202d4bb586c7c6b817158342 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3980e9202d4bb586c7c6b817158342; no voters: 2195f4b573b84fda85724f693eda2a36, cf99f2b9d85949f8ae4e5bbe8021050b
I20260810 04:09:20.747033 16215 raft_consensus.cc:2726] T 58fb54d17b9943cf8f21be2778663c72 P df3980e9202d4bb586c7c6b817158342 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 04:09:21.265936 16216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:21.265729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:44907 (local address 127.31.32.1:51527)
0810 04:09:21.265851 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0810 04:09:21.265887 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.62:44907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":66,"threads_started":1}
I20260810 04:09:21.357676 16217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:21.357448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:44907 (local address 127.31.32.2:54449)
0810 04:09:21.357596 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0810 04:09:21.357634 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.62:44907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":74,"threads_started":1}
I20260810 04:09:21.476694 16218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:21.476474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:44907 (local address 127.31.32.3:60355)
0810 04:09:21.476618 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0810 04:09:21.476654 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.62:44907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":79,"threads_started":1}
I20260810 04:09:22.266919 16219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:22.266500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:44907 (local address 127.31.32.1:54337)
0810 04:09:22.266798 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0810 04:09:22.266849 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.62:44907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":84,"threads_started":1}
I20260810 04:09:22.358335 16220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:22.358119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:44907 (local address 127.31.32.2:49963)
0810 04:09:22.358256 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0810 04:09:22.358291 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.62:44907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":67,"threads_started":1}
I20260810 04:09:22.477370 16221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:22.477149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:44907 (local address 127.31.32.3:38953)
0810 04:09:22.477282 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0810 04:09:22.477319 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.62:44907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":65,"threads_started":1}
I20260810 04:09:23.267663 16222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:23.267410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:44907 (local address 127.31.32.1:59149)
0810 04:09:23.267570 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0810 04:09:23.267613 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.62:44907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":70,"threads_started":1}
I20260810 04:09:23.359112 16223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:23.358843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:44907 (local address 127.31.32.2:41545)
0810 04:09:23.359017 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0810 04:09:23.359060 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.62:44907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":77,"threads_started":1}
I20260810 04:09:23.478202 16224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:23.477912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:44907 (local address 127.31.32.3:45807)
0810 04:09:23.478100 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0810 04:09:23.478144 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.62:44907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":94,"threads_started":1}
I20260810 04:09:24.268567 16225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:24.268340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:44907 (local address 127.31.32.1:39441)
0810 04:09:24.268480 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0810 04:09:24.268517 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.62:44907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":87,"threads_started":1}
I20260810 04:09:24.358106 16227 raft_consensus.cc:482] T 58fb54d17b9943cf8f21be2778663c72 P df3980e9202d4bb586c7c6b817158342 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 04:09:24.358219 16227 raft_consensus.cc:504] T 58fb54d17b9943cf8f21be2778663c72 P df3980e9202d4bb586c7c6b817158342 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2195f4b573b84fda85724f693eda2a36" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 46777 } } peers { permanent_uuid: "cf99f2b9d85949f8ae4e5bbe8021050b" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 34627 } } peers { permanent_uuid: "df3980e9202d4bb586c7c6b817158342" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 39207 } }
I20260810 04:09:24.358412 16227 leader_election.cc:288] T 58fb54d17b9943cf8f21be2778663c72 P df3980e9202d4bb586c7c6b817158342 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2195f4b573b84fda85724f693eda2a36 (127.31.32.3:46777), cf99f2b9d85949f8ae4e5bbe8021050b (127.31.32.2:34627)
W20260810 04:09:24.358817 15765 leader_election.cc:341] T 58fb54d17b9943cf8f21be2778663c72 P df3980e9202d4bb586c7c6b817158342 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2195f4b573b84fda85724f693eda2a36 (127.31.32.3:46777): Not found: Tablet not found: 58fb54d17b9943cf8f21be2778663c72
W20260810 04:09:24.358899 15762 leader_election.cc:341] T 58fb54d17b9943cf8f21be2778663c72 P df3980e9202d4bb586c7c6b817158342 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cf99f2b9d85949f8ae4e5bbe8021050b (127.31.32.2:34627): Not found: Tablet not found: 58fb54d17b9943cf8f21be2778663c72
I20260810 04:09:24.358960 15762 leader_election.cc:302] T 58fb54d17b9943cf8f21be2778663c72 P df3980e9202d4bb586c7c6b817158342 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3980e9202d4bb586c7c6b817158342; no voters: 2195f4b573b84fda85724f693eda2a36, cf99f2b9d85949f8ae4e5bbe8021050b
I20260810 04:09:24.359145 16227 raft_consensus.cc:2726] T 58fb54d17b9943cf8f21be2778663c72 P df3980e9202d4bb586c7c6b817158342 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 04:09:24.359766 16228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:24.359590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:44907 (local address 127.31.32.2:41783)
0810 04:09:24.359696 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0810 04:09:24.359727 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.62:44907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":49,"threads_started":1}
I20260810 04:09:24.478967 16229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:24.478717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:44907 (local address 127.31.32.3:47097)
0810 04:09:24.478881 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0810 04:09:24.478917 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.62:44907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":97,"threads_started":1}
I20260810 04:09:25.269265 16230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:25.269044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:44907 (local address 127.31.32.1:38687)
0810 04:09:25.269183 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0810 04:09:25.269218 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.62:44907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":64,"threads_started":1}
I20260810 04:09:25.360510 16231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:25.360256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:44907 (local address 127.31.32.2:57155)
0810 04:09:25.360432 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0810 04:09:25.360468 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.62:44907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":118,"threads_started":1}
I20260810 04:09:25.479676 16232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:25.479449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:44907 (local address 127.31.32.3:39303)
0810 04:09:25.479592 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0810 04:09:25.479631 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.62:44907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":77,"threads_started":1}
I20260810 04:09:26.269975 16233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:26.269755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:44907 (local address 127.31.32.1:57385)
0810 04:09:26.269888 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0810 04:09:26.269926 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.62:44907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":75,"threads_started":1}
I20260810 04:09:26.361434 16234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:26.361066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:44907 (local address 127.31.32.2:54631)
0810 04:09:26.361309 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0810 04:09:26.361390 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.62:44907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":65,"threads_started":1}
I20260810 04:09:26.480605 16235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:26.480371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:44907 (local address 127.31.32.3:51565)
0810 04:09:26.480518 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0810 04:09:26.480562 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.62:44907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":63,"threads_started":1}
I20260810 04:09:27.270875 16236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:27.270526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:44907 (local address 127.31.32.1:37857)
0810 04:09:27.270717 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0810 04:09:27.270779 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.62:44907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":74,"threads_started":1}
I20260810 04:09:27.362178 16237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:27.361962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:44907 (local address 127.31.32.2:45795)
0810 04:09:27.362100 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0810 04:09:27.362133 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.62:44907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":79,"threads_started":1}
I20260810 04:09:27.479511 31872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskws89Io/build/release/bin/kudu with pid 15714
I20260810 04:09:27.482184 16238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:27.481949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:44907 (local address 127.31.32.3:47321)
0810 04:09:27.482079 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0810 04:09:27.482132 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.62:44907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":64,"threads_started":1}
I20260810 04:09:27.486218 31872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskws89Io/build/release/bin/kudu with pid 15892
I20260810 04:09:27.494040 31872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskws89Io/build/release/bin/kudu with pid 16033
2026-08-10T04:09:27Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18124 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 04:09:27 dist-test-slave-8w9v krb5kdc[16241](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 04:09:27 dist-test-slave-8w9v krb5kdc[16241](info): set up 2 sockets
Aug 10 04:09:27 dist-test-slave-8w9v krb5kdc[16241](info): commencing operation
krb5kdc: starting...
W20260810 04:09:28.593940 31872 mini_kdc.cc:121] Time spent starting KDC: real 1.078s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/krb5kdc/test-user.keytab.
Aug 10 04:09:28 dist-test-slave-8w9v krb5kdc[16241](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334968, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T04:09:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T04:09:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.32.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.32.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.32.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.62.keytab.
Entry for principal kudu/127.31.32.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.62.keytab.
I20260810 04:09:28.714290 31872 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskws89Io/build/release/bin/kudu
/tmp/dist-test-taskws89Io/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.32.62:39943
--webserver_interface=127.31.32.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.62.keytab
--principal=kudu/127.31.32.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.32.20:40835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.32.62:39943
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:09:28.795904 16257 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:09:28.796186 16257 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:09:28.796255 16257 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:09:28.797429 16257 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260810 04:09:28.797492 16257 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260810 04:09:28.797514 16257 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260810 04:09:28.797533 16257 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:09:28.797564 16257 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:09:28.797591 16257 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:09:28.797614 16257 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:09:28.797645 16257 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:09:28.799171 16257 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.32.20:40835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.31.32.62:39943
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.62.keytab
--principal=kudu/127.31.32.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.32.62:39943
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.32.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16257
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:09:28.799461 16257 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:09:28.799721 16257 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:09:28.805214 16263 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:09:28.805217 16265 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:09:28.805229 16262 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:09:28.805274 16257 server_base.cc:1034] running on GCE node
I20260810 04:09:28.805742 16257 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:09:28.806130 16257 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:09:28.807318 16257 hybrid_clock.cc:648] HybridClock initialized: now 1786334968807295 us; error 34 us; skew 500 ppm
Aug 10 04:09:28 dist-test-slave-8w9v krb5kdc[16241](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334968, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:09:28.813781 16257 init.cc:377] Logged in from keytab as kudu/127.31.32.62@KRBTEST.COM (short username kudu)
I20260810 04:09:28.814271 16257 webserver.cc:460] Webserver started at http://127.31.32.62:42757/ using document root <none> and password file <none>
I20260810 04:09:28.814538 16257 fs_manager.cc:359] Metadata directory not provided
I20260810 04:09:28.814608 16257 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:09:28.814761 16257 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:09:28.815831 16257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/master-0/data/instance:
uuid: "39ad49896b2d4abf8bf3e221466c2cf5"
format_stamp: "Formatted at 2026-08-10 04:09:28 on dist-test-slave-8w9v"
server_key: "4465989c929bab8a58259d84eea563fe"
server_key_iv: "8cc3a42036dbb78e710b5de52047c8df"
server_key_version: "encryptionkey@0"
I20260810 04:09:28.816285 16257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/master-0/wal/instance:
uuid: "39ad49896b2d4abf8bf3e221466c2cf5"
format_stamp: "Formatted at 2026-08-10 04:09:28 on dist-test-slave-8w9v"
server_key: "4465989c929bab8a58259d84eea563fe"
server_key_iv: "8cc3a42036dbb78e710b5de52047c8df"
server_key_version: "encryptionkey@0"
I20260810 04:09:28.817619 16257 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260810 04:09:28.818387 16272 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:09:28.818567 16257 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260810 04:09:28.818667 16257 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/master-0/data,/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/master-0/wal
uuid: "39ad49896b2d4abf8bf3e221466c2cf5"
format_stamp: "Formatted at 2026-08-10 04:09:28 on dist-test-slave-8w9v"
server_key: "4465989c929bab8a58259d84eea563fe"
server_key_iv: "8cc3a42036dbb78e710b5de52047c8df"
server_key_version: "encryptionkey@0"
I20260810 04:09:28.818753 16257 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:09:28.833354 16257 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:09:28.835170 16257 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:09:28.835340 16257 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:09:28.839612 16257 rpc_server.cc:307] RPC server started. Bound to: 127.31.32.62:39943
I20260810 04:09:28.839645 16330 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.32.62:39943 every 8 connection(s)
I20260810 04:09:28.840006 16257 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/master-0/data/info.pb
I20260810 04:09:28.840970 16331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:09:28.843245 16331 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39ad49896b2d4abf8bf3e221466c2cf5: Bootstrap starting.
I20260810 04:09:28.843809 16331 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 39ad49896b2d4abf8bf3e221466c2cf5: Neither blocks nor log segments found. Creating new log.
I20260810 04:09:28.844085 16331 log.cc:824] T 00000000000000000000000000000000 P 39ad49896b2d4abf8bf3e221466c2cf5: Log is configured to *not* fsync() on all Append() calls
I20260810 04:09:28.844848 16331 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39ad49896b2d4abf8bf3e221466c2cf5: No bootstrap required, opened a new log
I20260810 04:09:28.846450 16331 raft_consensus.cc:348] T 00000000000000000000000000000000 P 39ad49896b2d4abf8bf3e221466c2cf5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39ad49896b2d4abf8bf3e221466c2cf5" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 39943 } }
I20260810 04:09:28.846590 16331 raft_consensus.cc:374] T 00000000000000000000000000000000 P 39ad49896b2d4abf8bf3e221466c2cf5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:09:28.846613 16331 raft_consensus.cc:729] T 00000000000000000000000000000000 P 39ad49896b2d4abf8bf3e221466c2cf5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39ad49896b2d4abf8bf3e221466c2cf5, State: Initialized, Role: FOLLOWER
I20260810 04:09:28.846773 16331 consensus_queue.cc:260] T 00000000000000000000000000000000 P 39ad49896b2d4abf8bf3e221466c2cf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39ad49896b2d4abf8bf3e221466c2cf5" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 39943 } }
I20260810 04:09:28.846846 16331 raft_consensus.cc:388] T 00000000000000000000000000000000 P 39ad49896b2d4abf8bf3e221466c2cf5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 04:09:28.846868 16331 raft_consensus.cc:482] T 00000000000000000000000000000000 P 39ad49896b2d4abf8bf3e221466c2cf5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 04:09:28.846936 16331 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 39ad49896b2d4abf8bf3e221466c2cf5 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:09:28.847530 16331 raft_consensus.cc:504] T 00000000000000000000000000000000 P 39ad49896b2d4abf8bf3e221466c2cf5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39ad49896b2d4abf8bf3e221466c2cf5" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 39943 } }
I20260810 04:09:28.847658 16331 leader_election.cc:302] T 00000000000000000000000000000000 P 39ad49896b2d4abf8bf3e221466c2cf5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 39ad49896b2d4abf8bf3e221466c2cf5; no voters:
I20260810 04:09:28.847843 16331 leader_election.cc:288] T 00000000000000000000000000000000 P 39ad49896b2d4abf8bf3e221466c2cf5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 04:09:28.847946 16334 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 39ad49896b2d4abf8bf3e221466c2cf5 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:09:28.848181 16334 raft_consensus.cc:686] T 00000000000000000000000000000000 P 39ad49896b2d4abf8bf3e221466c2cf5 [term 1 LEADER]: Becoming Leader. State: Replica: 39ad49896b2d4abf8bf3e221466c2cf5, State: Running, Role: LEADER
I20260810 04:09:28.848217 16331 sys_catalog.cc:564] T 00000000000000000000000000000000 P 39ad49896b2d4abf8bf3e221466c2cf5 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:09:28.848286 16334 consensus_queue.cc:237] T 00000000000000000000000000000000 P 39ad49896b2d4abf8bf3e221466c2cf5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39ad49896b2d4abf8bf3e221466c2cf5" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 39943 } }
I20260810 04:09:28.848716 16335 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39ad49896b2d4abf8bf3e221466c2cf5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "39ad49896b2d4abf8bf3e221466c2cf5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39ad49896b2d4abf8bf3e221466c2cf5" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 39943 } } }
I20260810 04:09:28.848891 31872 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskws89Io/build/release/bin/kudu as pid 16257
I20260810 04:09:28.849001 31872 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/master-0/wal/instance
I20260810 04:09:28.849017 16335 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39ad49896b2d4abf8bf3e221466c2cf5 [sys.catalog]: This master's current role is: LEADER
I20260810 04:09:28.849262 31872 external_mini_cluster.cc:1428] Setting key 6e4fb2b6b8b181a0720fb7aec48f49d4
I20260810 04:09:28.848939 16336 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39ad49896b2d4abf8bf3e221466c2cf5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 39ad49896b2d4abf8bf3e221466c2cf5. Latest consensus state: current_term: 1 leader_uuid: "39ad49896b2d4abf8bf3e221466c2cf5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39ad49896b2d4abf8bf3e221466c2cf5" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 39943 } } }
I20260810 04:09:28.849424 16336 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39ad49896b2d4abf8bf3e221466c2cf5 [sys.catalog]: This master's current role is: LEADER
I20260810 04:09:28.849529 16343 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 04:09:28.850106 16343 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 04:09:28.851738 16343 catalog_manager.cc:1347] Generated new cluster ID: 0bedc7d1051749e099934262464f9bde
I20260810 04:09:28.851790 16343 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 10 04:09:28 dist-test-slave-8w9v krb5kdc[16241](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334968, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.32.62@KRBTEST.COM
I20260810 04:09:28.861096 16343 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 04:09:28.861864 16343 catalog_manager.cc:1504] Loading token signing keys...
I20260810 04:09:28.862110 16353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:28.849973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35908 (local address 127.31.32.62:39943)
0810 04:09:28.850159 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0810 04:09:28.850170 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 04:09:28.850205 (+ 35us) server_negotiation.cc:407] Connection header received
0810 04:09:28.850338 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:09:28.850345 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:09:28.850403 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:09:28.850548 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:09:28.851358 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:09:28.852772 (+ 1414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:09:28.853913 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:09:28.854544 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:09:28.854966 (+ 422us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:09:28.857591 (+ 2625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:09:28.857596 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:09:28.857596 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:09:28.857609 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:09:28.860312 (+ 2703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:09:28.860740 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:09:28.860743 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:09:28.860744 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:09:28.860854 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:09:28.861095 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:09:28.861097 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:09:28.861097 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:09:28.861410 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:09:28.861557 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0810 04:09:28.861718 (+ 161us) server_negotiation.cc:299] Negotiation successful
0810 04:09:28.861891 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":84,"threads_started":1}
I20260810 04:09:28.869453 16343 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 39ad49896b2d4abf8bf3e221466c2cf5: Generated new TSK 0
I20260810 04:09:28.869655 16343 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.32.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.32.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.32.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.1.keytab.
Entry for principal kudu/127.31.32.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.1.keytab.
I20260810 04:09:28.903051 31872 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskws89Io/build/release/bin/kudu
/tmp/dist-test-taskws89Io/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.32.1:0
--local_ip_for_outbound_sockets=127.31.32.1
--webserver_interface=127.31.32.1
--webserver_port=0
--tserver_master_addrs=127.31.32.62:39943
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.1.keytab
--principal=kudu/127.31.32.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.32.20:40835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:09:28.987102 16358 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:09:28.987294 16358 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:09:28.987316 16358 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:09:28.988508 16358 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:09:28.988548 16358 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:09:28.988574 16358 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.32.1
I20260810 04:09:28.989923 16358 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.32.20:40835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.1.keytab
--principal=kudu/127.31.32.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.32.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.32.1
--webserver_port=0
--tserver_master_addrs=127.31.32.62:39943
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16358
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.32.1
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:09:28.990095 16358 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:09:28.990264 16358 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:09:28.995254 16367 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:09:28.995235 16364 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:09:28.995235 16363 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:09:28.995337 16358 server_base.cc:1034] running on GCE node
I20260810 04:09:28.995723 16358 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:09:28.995967 16358 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:09:28.997143 16358 hybrid_clock.cc:648] HybridClock initialized: now 1786334968997142 us; error 52 us; skew 500 ppm
Aug 10 04:09:28 dist-test-slave-8w9v krb5kdc[16241](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334968, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:09:28.999776 16358 init.cc:377] Logged in from keytab as kudu/127.31.32.1@KRBTEST.COM (short username kudu)
I20260810 04:09:29.000317 16358 webserver.cc:460] Webserver started at http://127.31.32.1:45745/ using document root <none> and password file <none>
I20260810 04:09:29.000576 16358 fs_manager.cc:359] Metadata directory not provided
I20260810 04:09:29.000645 16358 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:09:29.000792 16358 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:09:29.001845 16358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/ts-0/data/instance:
uuid: "7b0dea1b85284b918eb1601d887ae01b"
format_stamp: "Formatted at 2026-08-10 04:09:28 on dist-test-slave-8w9v"
server_key: "b5b53b670d118fb11fea1357903cd359"
server_key_iv: "a2b1e2ab3dd73110d77044d83054bdc6"
server_key_version: "encryptionkey@0"
I20260810 04:09:29.002228 16358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/ts-0/wal/instance:
uuid: "7b0dea1b85284b918eb1601d887ae01b"
format_stamp: "Formatted at 2026-08-10 04:09:28 on dist-test-slave-8w9v"
server_key: "b5b53b670d118fb11fea1357903cd359"
server_key_iv: "a2b1e2ab3dd73110d77044d83054bdc6"
server_key_version: "encryptionkey@0"
I20260810 04:09:29.003679 16358 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 04:09:29.004591 16374 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:09:29.004778 16358 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260810 04:09:29.004876 16358 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/ts-0/data,/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/ts-0/wal
uuid: "7b0dea1b85284b918eb1601d887ae01b"
format_stamp: "Formatted at 2026-08-10 04:09:28 on dist-test-slave-8w9v"
server_key: "b5b53b670d118fb11fea1357903cd359"
server_key_iv: "a2b1e2ab3dd73110d77044d83054bdc6"
server_key_version: "encryptionkey@0"
I20260810 04:09:29.004963 16358 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:09:29.022177 16358 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:09:29.024638 16358 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:09:29.024814 16358 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:09:29.025063 16358 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:09:29.025403 16358 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:09:29.025461 16358 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:09:29.025503 16358 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:09:29.025532 16358 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:09:29.031684 16358 rpc_server.cc:307] RPC server started. Bound to: 127.31.32.1:46861
I20260810 04:09:29.031752 16496 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.32.1:46861 every 8 connection(s)
I20260810 04:09:29.032106 16358 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/ts-0/data/info.pb
Aug 10 04:09:29 dist-test-slave-8w9v krb5kdc[16241](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334968, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.1@KRBTEST.COM for kudu/127.31.32.62@KRBTEST.COM
I20260810 04:09:29.038740 31872 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskws89Io/build/release/bin/kudu as pid 16358
I20260810 04:09:29.038867 31872 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/ts-0/wal/instance
I20260810 04:09:29.039096 31872 external_mini_cluster.cc:1428] Setting key 9f9f114d273ba59b35c0397dba16f973
I20260810 04:09:29.042173 16353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:29.032953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.1:40697 (local address 127.31.32.62:39943)
0810 04:09:29.033015 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0810 04:09:29.033017 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:09:29.033252 (+ 235us) server_negotiation.cc:407] Connection header received
0810 04:09:29.033815 (+ 563us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:09:29.033816 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:09:29.033827 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:09:29.033870 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:09:29.034922 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:09:29.035457 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:09:29.036651 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:09:29.036870 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:09:29.036932 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:09:29.039232 (+ 2300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:09:29.039234 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:09:29.039234 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:09:29.039248 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:09:29.040761 (+ 1513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:09:29.041184 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:09:29.041185 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:09:29.041186 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:09:29.041246 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:09:29.041507 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:09:29.041508 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:09:29.041509 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:09:29.041620 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:09:29.041691 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0810 04:09:29.041981 (+ 290us) server_negotiation.cc:299] Negotiation successful
0810 04:09:29.042077 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260810 04:09:29.042380 16499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:29.032929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:39943 (local address 127.31.32.1:40697)
0810 04:09:29.033113 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0810 04:09:29.033159 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0810 04:09:29.033706 (+ 547us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:09:29.033939 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:09:29.033943 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:09:29.034607 (+ 664us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:09:29.034820 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:09:29.034826 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:09:29.035551 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:09:29.035552 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:09:29.036558 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:09:29.036561 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:09:29.036964 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:09:29.036965 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:09:29.037077 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:09:29.037549 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:09:29.037558 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:09:29.039143 (+ 1585us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:09:29.040850 (+ 1707us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:09:29.040858 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:09:29.040861 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:09:29.041100 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:09:29.041329 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:09:29.041330 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:09:29.041330 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:09:29.041439 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:09:29.041696 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:09:29.041704 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:09:29.041843 (+ 139us) client_negotiation.cc:769] Sending connection context
0810 04:09:29.041957 (+ 114us) client_negotiation.cc:240] Negotiation successful
0810 04:09:29.042126 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":106,"threads_started":1}
I20260810 04:09:29.042877 16497 heartbeater.cc:347] Connected to a master server at 127.31.32.62:39943
I20260810 04:09:29.042977 16497 heartbeater.cc:464] Registering TS with master...
I20260810 04:09:29.043237 16497 heartbeater.cc:511] Master 127.31.32.62:39943 requested a full tablet report, sending...
I20260810 04:09:29.043908 16291 ts_manager.cc:194] Registered new tserver with Master: 7b0dea1b85284b918eb1601d887ae01b (127.31.32.1:46861)
I20260810 04:09:29.044916 16291 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.32.1@KRBTEST.COM'} at 127.31.32.1:40697
WARNING: no policy specified for kudu/127.31.32.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.32.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.32.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.2.keytab.
Entry for principal kudu/127.31.32.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.2.keytab.
I20260810 04:09:29.065145 31872 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskws89Io/build/release/bin/kudu
/tmp/dist-test-taskws89Io/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.32.2:0
--local_ip_for_outbound_sockets=127.31.32.2
--webserver_interface=127.31.32.2
--webserver_port=0
--tserver_master_addrs=127.31.32.62:39943
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.2.keytab
--principal=kudu/127.31.32.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.32.20:40835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:09:29.147220 16503 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:09:29.147416 16503 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:09:29.147439 16503 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:09:29.148677 16503 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:09:29.148718 16503 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:09:29.148746 16503 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.32.2
I20260810 04:09:29.150264 16503 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.32.20:40835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.2.keytab
--principal=kudu/127.31.32.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.32.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.32.2
--webserver_port=0
--tserver_master_addrs=127.31.32.62:39943
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16503
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.32.2
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:09:29.150434 16503 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:09:29.150590 16503 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:09:29.155406 16511 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:09:29.155457 16508 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:09:29.155417 16509 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:09:29.155587 16503 server_base.cc:1034] running on GCE node
I20260810 04:09:29.155843 16503 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:09:29.156173 16503 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:09:29.157357 16503 hybrid_clock.cc:648] HybridClock initialized: now 1786334969157345 us; error 33 us; skew 500 ppm
Aug 10 04:09:29 dist-test-slave-8w9v krb5kdc[16241](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334969, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:09:29.159932 16503 init.cc:377] Logged in from keytab as kudu/127.31.32.2@KRBTEST.COM (short username kudu)
I20260810 04:09:29.160434 16503 webserver.cc:460] Webserver started at http://127.31.32.2:39677/ using document root <none> and password file <none>
I20260810 04:09:29.160657 16503 fs_manager.cc:359] Metadata directory not provided
I20260810 04:09:29.160725 16503 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:09:29.160888 16503 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:09:29.161885 16503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/ts-1/data/instance:
uuid: "1fa692fb508741e1be5992f3fd536b48"
format_stamp: "Formatted at 2026-08-10 04:09:29 on dist-test-slave-8w9v"
server_key: "64a1f9ecbd6b76abc5e22845cee6b701"
server_key_iv: "8518bbb579c2bd18ac4604451e2b4cc5"
server_key_version: "encryptionkey@0"
I20260810 04:09:29.162242 16503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/ts-1/wal/instance:
uuid: "1fa692fb508741e1be5992f3fd536b48"
format_stamp: "Formatted at 2026-08-10 04:09:29 on dist-test-slave-8w9v"
server_key: "64a1f9ecbd6b76abc5e22845cee6b701"
server_key_iv: "8518bbb579c2bd18ac4604451e2b4cc5"
server_key_version: "encryptionkey@0"
I20260810 04:09:29.163568 16503 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260810 04:09:29.164433 16518 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:09:29.164628 16503 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:09:29.164727 16503 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/ts-1/data,/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/ts-1/wal
uuid: "1fa692fb508741e1be5992f3fd536b48"
format_stamp: "Formatted at 2026-08-10 04:09:29 on dist-test-slave-8w9v"
server_key: "64a1f9ecbd6b76abc5e22845cee6b701"
server_key_iv: "8518bbb579c2bd18ac4604451e2b4cc5"
server_key_version: "encryptionkey@0"
I20260810 04:09:29.164822 16503 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:09:29.178519 16503 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:09:29.180451 16503 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:09:29.180617 16503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:09:29.180850 16503 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:09:29.181155 16503 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:09:29.181210 16503 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:09:29.181249 16503 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:09:29.181277 16503 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:09:29.187055 16503 rpc_server.cc:307] RPC server started. Bound to: 127.31.32.2:45549
I20260810 04:09:29.187107 16644 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.32.2:45549 every 8 connection(s)
I20260810 04:09:29.187407 16503 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/ts-1/data/info.pb
I20260810 04:09:29.190049 31872 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskws89Io/build/release/bin/kudu as pid 16503
I20260810 04:09:29.190181 31872 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/ts-1/wal/instance
I20260810 04:09:29.190405 31872 external_mini_cluster.cc:1428] Setting key 4e8bd3c697415c81efc8026fe4cc9d2b
Aug 10 04:09:29 dist-test-slave-8w9v krb5kdc[16241](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334969, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.2@KRBTEST.COM for kudu/127.31.32.62@KRBTEST.COM
I20260810 04:09:29.197296 16353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:29.188074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.2:34363 (local address 127.31.32.62:39943)
0810 04:09:29.188183 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0810 04:09:29.188185 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:09:29.188329 (+ 144us) server_negotiation.cc:407] Connection header received
0810 04:09:29.188871 (+ 542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:09:29.188872 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:09:29.188885 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:09:29.189120 (+ 235us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:09:29.190075 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:09:29.190730 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:09:29.191790 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:09:29.192018 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:09:29.192106 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:09:29.194356 (+ 2250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:09:29.194358 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:09:29.194358 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:09:29.194364 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:09:29.195795 (+ 1431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:09:29.196337 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:09:29.196338 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:09:29.196338 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:09:29.196388 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:09:29.196679 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:09:29.196680 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:09:29.196680 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:09:29.196774 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:09:29.196822 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0810 04:09:29.197165 (+ 343us) server_negotiation.cc:299] Negotiation successful
0810 04:09:29.197234 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260810 04:09:29.197583 16647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:29.188023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:39943 (local address 127.31.32.2:34363)
0810 04:09:29.188207 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0810 04:09:29.188247 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0810 04:09:29.188720 (+ 473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:09:29.189202 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:09:29.189206 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:09:29.189766 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:09:29.189977 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:09:29.189982 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:09:29.190813 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:09:29.190814 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:09:29.191685 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:09:29.191688 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:09:29.192137 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:09:29.192138 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:09:29.192268 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:09:29.192789 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:09:29.192798 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:09:29.194264 (+ 1466us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:09:29.195881 (+ 1617us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:09:29.195889 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:09:29.195891 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:09:29.196259 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:09:29.196460 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:09:29.196461 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:09:29.196462 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:09:29.196608 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:09:29.196844 (+ 236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:09:29.196850 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:09:29.197019 (+ 169us) client_negotiation.cc:769] Sending connection context
0810 04:09:29.197137 (+ 118us) client_negotiation.cc:240] Negotiation successful
0810 04:09:29.197313 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":114,"threads_started":1}
I20260810 04:09:29.198148 16645 heartbeater.cc:347] Connected to a master server at 127.31.32.62:39943
I20260810 04:09:29.198257 16645 heartbeater.cc:464] Registering TS with master...
I20260810 04:09:29.198432 16645 heartbeater.cc:511] Master 127.31.32.62:39943 requested a full tablet report, sending...
I20260810 04:09:29.198827 16291 ts_manager.cc:194] Registered new tserver with Master: 1fa692fb508741e1be5992f3fd536b48 (127.31.32.2:45549)
I20260810 04:09:29.199193 16291 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.32.2@KRBTEST.COM'} at 127.31.32.2:34363
WARNING: no policy specified for kudu/127.31.32.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.32.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.32.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.3.keytab.
Entry for principal kudu/127.31.32.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.3.keytab.
I20260810 04:09:29.216996 31872 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskws89Io/build/release/bin/kudu
/tmp/dist-test-taskws89Io/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.32.3:0
--local_ip_for_outbound_sockets=127.31.32.3
--webserver_interface=127.31.32.3
--webserver_port=0
--tserver_master_addrs=127.31.32.62:39943
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.3.keytab
--principal=kudu/127.31.32.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.32.20:40835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:09:29.299167 16650 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:09:29.299367 16650 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:09:29.299391 16650 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:09:29.300549 16650 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:09:29.300587 16650 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:09:29.300638 16650 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.32.3
I20260810 04:09:29.302040 16650 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.32.20:40835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.3.keytab
--principal=kudu/127.31.32.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.32.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.32.3
--webserver_port=0
--tserver_master_addrs=127.31.32.62:39943
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16650
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.32.3
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:09:29.302282 16650 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:09:29.302515 16650 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:09:29.307507 16658 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:09:29.307518 16655 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:09:29.307508 16656 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:09:29.307534 16650 server_base.cc:1034] running on GCE node
I20260810 04:09:29.308022 16650 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:09:29.308346 16650 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:09:29.309507 16650 hybrid_clock.cc:648] HybridClock initialized: now 1786334969309497 us; error 22 us; skew 500 ppm
Aug 10 04:09:29 dist-test-slave-8w9v krb5kdc[16241](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334969, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:09:29.311945 16650 init.cc:377] Logged in from keytab as kudu/127.31.32.3@KRBTEST.COM (short username kudu)
I20260810 04:09:29.312417 16650 webserver.cc:460] Webserver started at http://127.31.32.3:43127/ using document root <none> and password file <none>
I20260810 04:09:29.312642 16650 fs_manager.cc:359] Metadata directory not provided
I20260810 04:09:29.312708 16650 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:09:29.312839 16650 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:09:29.313836 16650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/ts-2/data/instance:
uuid: "9eda5e08e6434319b8b3626b9fabf245"
format_stamp: "Formatted at 2026-08-10 04:09:29 on dist-test-slave-8w9v"
server_key: "7b9b99a18e68ebcd40b887bca207e8fd"
server_key_iv: "d591e51a46ccd8da63898fbdcd65317b"
server_key_version: "encryptionkey@0"
I20260810 04:09:29.314198 16650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/ts-2/wal/instance:
uuid: "9eda5e08e6434319b8b3626b9fabf245"
format_stamp: "Formatted at 2026-08-10 04:09:29 on dist-test-slave-8w9v"
server_key: "7b9b99a18e68ebcd40b887bca207e8fd"
server_key_iv: "d591e51a46ccd8da63898fbdcd65317b"
server_key_version: "encryptionkey@0"
I20260810 04:09:29.315503 16650 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 04:09:29.316277 16665 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:09:29.316459 16650 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 04:09:29.316560 16650 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/ts-2/data,/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/ts-2/wal
uuid: "9eda5e08e6434319b8b3626b9fabf245"
format_stamp: "Formatted at 2026-08-10 04:09:29 on dist-test-slave-8w9v"
server_key: "7b9b99a18e68ebcd40b887bca207e8fd"
server_key_iv: "d591e51a46ccd8da63898fbdcd65317b"
server_key_version: "encryptionkey@0"
I20260810 04:09:29.316633 16650 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:09:29.330665 16650 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:09:29.332441 16650 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:09:29.332592 16650 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:09:29.332808 16650 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:09:29.333110 16650 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:09:29.333163 16650 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:09:29.333204 16650 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:09:29.333232 16650 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:09:29.339017 16650 rpc_server.cc:307] RPC server started. Bound to: 127.31.32.3:44889
I20260810 04:09:29.339069 16791 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.32.3:44889 every 8 connection(s)
I20260810 04:09:29.339366 16650 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/ts-2/data/info.pb
I20260810 04:09:29.341634 31872 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskws89Io/build/release/bin/kudu as pid 16650
I20260810 04:09:29.341794 31872 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/ts-2/wal/instance
I20260810 04:09:29.342070 31872 external_mini_cluster.cc:1428] Setting key 51b1b38ba442c1e76a92ad96882dc2d7
Aug 10 04:09:29 dist-test-slave-8w9v krb5kdc[16241](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334969, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.3@KRBTEST.COM for kudu/127.31.32.62@KRBTEST.COM
I20260810 04:09:29.348784 16353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:29.340076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.3:51625 (local address 127.31.32.62:39943)
0810 04:09:29.340158 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0810 04:09:29.340160 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:09:29.340387 (+ 227us) server_negotiation.cc:407] Connection header received
0810 04:09:29.340889 (+ 502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:09:29.340890 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:09:29.340903 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:09:29.340969 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:09:29.341972 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:09:29.342738 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:09:29.343729 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:09:29.343923 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:09:29.343979 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:09:29.345993 (+ 2014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:09:29.345994 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:09:29.345994 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:09:29.346001 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:09:29.347431 (+ 1430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:09:29.347846 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:09:29.347847 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:09:29.347847 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:09:29.347901 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:09:29.348116 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:09:29.348117 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:09:29.348117 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:09:29.348260 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:09:29.348314 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0810 04:09:29.348650 (+ 336us) server_negotiation.cc:299] Negotiation successful
0810 04:09:29.348719 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260810 04:09:29.349073 16794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:29.340011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:39943 (local address 127.31.32.3:51625)
0810 04:09:29.340232 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0810 04:09:29.340291 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0810 04:09:29.340762 (+ 471us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:09:29.341044 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:09:29.341047 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:09:29.341593 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:09:29.341801 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:09:29.341807 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:09:29.342854 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:09:29.342855 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:09:29.343654 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:09:29.343656 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:09:29.343998 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:09:29.343999 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:09:29.344095 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:09:29.344569 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:09:29.344576 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:09:29.345908 (+ 1332us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:09:29.347512 (+ 1604us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:09:29.347519 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:09:29.347520 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:09:29.347765 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:09:29.347962 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:09:29.347963 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:09:29.347963 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:09:29.348060 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:09:29.348319 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:09:29.348324 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:09:29.348513 (+ 189us) client_negotiation.cc:769] Sending connection context
0810 04:09:29.348631 (+ 118us) client_negotiation.cc:240] Negotiation successful
0810 04:09:29.348770 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":70,"threads_started":1}
I20260810 04:09:29.349615 16792 heartbeater.cc:347] Connected to a master server at 127.31.32.62:39943
I20260810 04:09:29.349722 16792 heartbeater.cc:464] Registering TS with master...
I20260810 04:09:29.349931 16792 heartbeater.cc:511] Master 127.31.32.62:39943 requested a full tablet report, sending...
I20260810 04:09:29.350340 16291 ts_manager.cc:194] Registered new tserver with Master: 9eda5e08e6434319b8b3626b9fabf245 (127.31.32.3:44889)
I20260810 04:09:29.350723 16291 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.32.3@KRBTEST.COM'} at 127.31.32.3:51625
I20260810 04:09:29.353885 31872 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 04:09:29.361212 16353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:29.355269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35912 (local address 127.31.32.62:39943)
0810 04:09:29.355345 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0810 04:09:29.355347 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:09:29.355456 (+ 109us) server_negotiation.cc:407] Connection header received
0810 04:09:29.355533 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:09:29.355534 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:09:29.355544 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:09:29.355599 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:09:29.356211 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:09:29.356675 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:09:29.357416 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:09:29.357605 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:09:29.357676 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:09:29.358846 (+ 1170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:09:29.358848 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:09:29.358848 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:09:29.358855 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:09:29.359982 (+ 1127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:09:29.360383 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:09:29.360384 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:09:29.360384 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:09:29.360440 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:09:29.360659 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:09:29.360659 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:09:29.360660 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:09:29.360812 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:09:29.360919 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0810 04:09:29.361041 (+ 122us) server_negotiation.cc:299] Negotiation successful
0810 04:09:29.361116 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260810 04:09:29.362804 16291 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35912:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 04:09:29.364348 16291 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 04:09:29.371009 16802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:29.366551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43686 (local address 127.31.32.3:44889)
0810 04:09:29.367269 (+ 718us) server_negotiation.cc:206] Beginning negotiation
0810 04:09:29.367273 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 04:09:29.367316 (+ 43us) server_negotiation.cc:407] Connection header received
0810 04:09:29.367353 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:09:29.367355 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:09:29.367370 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:09:29.367441 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:09:29.367637 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:09:29.368574 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:09:29.370020 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:09:29.370683 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:09:29.370744 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:09:29.370792 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0810 04:09:29.370890 (+ 98us) server_negotiation.cc:299] Negotiation successful
0810 04:09:29.370926 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":677,"thread_start_us":83,"threads_started":1}
I20260810 04:09:29.371022 16801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:29.366623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.2:45549 (local address 127.0.0.1:45036)
0810 04:09:29.366817 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0810 04:09:29.366838 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 04:09:29.366886 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:09:29.367237 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:09:29.367238 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:09:29.367240 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:09:29.367296 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:09:29.367297 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:09:29.368572 (+ 1275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:09:29.368573 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:09:29.370047 (+ 1474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:09:29.370049 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:09:29.370796 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:09:29.370797 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:09:29.370851 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:09:29.370854 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 04:09:29.370902 (+ 48us) client_negotiation.cc:240] Negotiation successful
0810 04:09:29.370941 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"mutex_wait_us":87,"thread_start_us":44,"threads_started":1}
I20260810 04:09:29.371106 16804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:29.366637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52852 (local address 127.31.32.1:46861)
0810 04:09:29.366820 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0810 04:09:29.366826 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 04:09:29.366863 (+ 37us) server_negotiation.cc:407] Connection header received
0810 04:09:29.367001 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:09:29.367004 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:09:29.367018 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:09:29.367122 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:09:29.367306 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:09:29.368469 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:09:29.369814 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:09:29.370516 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:09:29.370593 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:09:29.370649 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0810 04:09:29.370969 (+ 320us) server_negotiation.cc:299] Negotiation successful
0810 04:09:29.371016 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":86,"threads_started":1}
I20260810 04:09:29.371249 16805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:29.366816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45036 (local address 127.31.32.2:45549)
0810 04:09:29.366987 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0810 04:09:29.366991 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 04:09:29.366999 (+ 8us) server_negotiation.cc:407] Connection header received
0810 04:09:29.367042 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:09:29.367044 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:09:29.367056 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:09:29.367157 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:09:29.367376 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:09:29.368469 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:09:29.370130 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:09:29.370633 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:09:29.370702 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:09:29.370761 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0810 04:09:29.371134 (+ 373us) server_negotiation.cc:299] Negotiation successful
0810 04:09:29.371184 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":94,"threads_started":1}
I20260810 04:09:29.371027 16800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:29.366425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.3:44889 (local address 127.0.0.1:43686)
0810 04:09:29.366523 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0810 04:09:29.366559 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0810 04:09:29.366627 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:09:29.367515 (+ 888us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:09:29.367515 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:09:29.367517 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:09:29.367560 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:09:29.367562 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:09:29.368668 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:09:29.368668 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:09:29.369933 (+ 1265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:09:29.369936 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:09:29.370747 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:09:29.370748 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:09:29.370813 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:09:29.370818 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 04:09:29.370877 (+ 59us) client_negotiation.cc:240] Negotiation successful
0810 04:09:29.370918 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":74,"mutex_wait_us":39,"thread_start_us":56,"threads_started":1}
I20260810 04:09:29.371827 16803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:29.366427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.1:46861 (local address 127.0.0.1:52852)
0810 04:09:29.366798 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0810 04:09:29.366809 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 04:09:29.366911 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:09:29.367134 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:09:29.367137 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:09:29.367139 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:09:29.367203 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:09:29.367205 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:09:29.368797 (+ 1592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:09:29.368798 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:09:29.369725 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:09:29.369727 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:09:29.370609 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:09:29.370610 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:09:29.370706 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:09:29.370721 (+ 15us) client_negotiation.cc:769] Sending connection context
0810 04:09:29.370787 (+ 66us) client_negotiation.cc:240] Negotiation successful
0810 04:09:29.370849 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":51,"threads_started":1}
I20260810 04:09:29.372328 16570 tablet_service.cc:1467] Processing CreateTablet for tablet b0c0666e4e774091b0edd8ae5af8ef09 (DEFAULT_TABLE table=test-table [id=5e452ca7c268412c86e30c97968f5f94]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:09:29.372570 16718 tablet_service.cc:1467] Processing CreateTablet for tablet b0c0666e4e774091b0edd8ae5af8ef09 (DEFAULT_TABLE table=test-table [id=5e452ca7c268412c86e30c97968f5f94]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:09:29.372681 16570 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0c0666e4e774091b0edd8ae5af8ef09. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:09:29.372895 16718 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0c0666e4e774091b0edd8ae5af8ef09. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:09:29.375466 16806 tablet_bootstrap.cc:492] T b0c0666e4e774091b0edd8ae5af8ef09 P 1fa692fb508741e1be5992f3fd536b48: Bootstrap starting.
I20260810 04:09:29.375484 16807 tablet_bootstrap.cc:492] T b0c0666e4e774091b0edd8ae5af8ef09 P 9eda5e08e6434319b8b3626b9fabf245: Bootstrap starting.
I20260810 04:09:29.375916 16423 tablet_service.cc:1467] Processing CreateTablet for tablet b0c0666e4e774091b0edd8ae5af8ef09 (DEFAULT_TABLE table=test-table [id=5e452ca7c268412c86e30c97968f5f94]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:09:29.376117 16806 tablet_bootstrap.cc:654] T b0c0666e4e774091b0edd8ae5af8ef09 P 1fa692fb508741e1be5992f3fd536b48: Neither blocks nor log segments found. Creating new log.
I20260810 04:09:29.376190 16807 tablet_bootstrap.cc:654] T b0c0666e4e774091b0edd8ae5af8ef09 P 9eda5e08e6434319b8b3626b9fabf245: Neither blocks nor log segments found. Creating new log.
I20260810 04:09:29.376279 16423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0c0666e4e774091b0edd8ae5af8ef09. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:09:29.376688 16806 log.cc:824] T b0c0666e4e774091b0edd8ae5af8ef09 P 1fa692fb508741e1be5992f3fd536b48: Log is configured to *not* fsync() on all Append() calls
I20260810 04:09:29.376748 16807 log.cc:824] T b0c0666e4e774091b0edd8ae5af8ef09 P 9eda5e08e6434319b8b3626b9fabf245: Log is configured to *not* fsync() on all Append() calls
I20260810 04:09:29.377511 16806 tablet_bootstrap.cc:492] T b0c0666e4e774091b0edd8ae5af8ef09 P 1fa692fb508741e1be5992f3fd536b48: No bootstrap required, opened a new log
I20260810 04:09:29.377590 16806 ts_tablet_manager.cc:1397] T b0c0666e4e774091b0edd8ae5af8ef09 P 1fa692fb508741e1be5992f3fd536b48: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 04:09:29.379535 16810 tablet_bootstrap.cc:492] T b0c0666e4e774091b0edd8ae5af8ef09 P 7b0dea1b85284b918eb1601d887ae01b: Bootstrap starting.
I20260810 04:09:29.379720 16806 raft_consensus.cc:348] T b0c0666e4e774091b0edd8ae5af8ef09 P 1fa692fb508741e1be5992f3fd536b48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9eda5e08e6434319b8b3626b9fabf245" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 44889 } } peers { permanent_uuid: "7b0dea1b85284b918eb1601d887ae01b" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 46861 } } peers { permanent_uuid: "1fa692fb508741e1be5992f3fd536b48" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 45549 } }
I20260810 04:09:29.379913 16806 raft_consensus.cc:374] T b0c0666e4e774091b0edd8ae5af8ef09 P 1fa692fb508741e1be5992f3fd536b48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:09:29.379969 16806 raft_consensus.cc:729] T b0c0666e4e774091b0edd8ae5af8ef09 P 1fa692fb508741e1be5992f3fd536b48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fa692fb508741e1be5992f3fd536b48, State: Initialized, Role: FOLLOWER
I20260810 04:09:29.380091 16806 consensus_queue.cc:260] T b0c0666e4e774091b0edd8ae5af8ef09 P 1fa692fb508741e1be5992f3fd536b48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9eda5e08e6434319b8b3626b9fabf245" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 44889 } } peers { permanent_uuid: "7b0dea1b85284b918eb1601d887ae01b" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 46861 } } peers { permanent_uuid: "1fa692fb508741e1be5992f3fd536b48" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 45549 } }
I20260810 04:09:29.380285 16810 tablet_bootstrap.cc:654] T b0c0666e4e774091b0edd8ae5af8ef09 P 7b0dea1b85284b918eb1601d887ae01b: Neither blocks nor log segments found. Creating new log.
I20260810 04:09:29.380448 16806 ts_tablet_manager.cc:1428] T b0c0666e4e774091b0edd8ae5af8ef09 P 1fa692fb508741e1be5992f3fd536b48: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 04:09:29.380492 16645 heartbeater.cc:503] Master 127.31.32.62:39943 was elected leader, sending a full tablet report...
I20260810 04:09:29.380514 16807 tablet_bootstrap.cc:492] T b0c0666e4e774091b0edd8ae5af8ef09 P 9eda5e08e6434319b8b3626b9fabf245: No bootstrap required, opened a new log
I20260810 04:09:29.380569 16807 ts_tablet_manager.cc:1397] T b0c0666e4e774091b0edd8ae5af8ef09 P 9eda5e08e6434319b8b3626b9fabf245: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20260810 04:09:29.381184 16810 log.cc:824] T b0c0666e4e774091b0edd8ae5af8ef09 P 7b0dea1b85284b918eb1601d887ae01b: Log is configured to *not* fsync() on all Append() calls
I20260810 04:09:29.381989 16810 tablet_bootstrap.cc:492] T b0c0666e4e774091b0edd8ae5af8ef09 P 7b0dea1b85284b918eb1601d887ae01b: No bootstrap required, opened a new log
I20260810 04:09:29.382042 16810 ts_tablet_manager.cc:1397] T b0c0666e4e774091b0edd8ae5af8ef09 P 7b0dea1b85284b918eb1601d887ae01b: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260810 04:09:29.382216 16807 raft_consensus.cc:348] T b0c0666e4e774091b0edd8ae5af8ef09 P 9eda5e08e6434319b8b3626b9fabf245 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9eda5e08e6434319b8b3626b9fabf245" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 44889 } } peers { permanent_uuid: "7b0dea1b85284b918eb1601d887ae01b" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 46861 } } peers { permanent_uuid: "1fa692fb508741e1be5992f3fd536b48" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 45549 } }
I20260810 04:09:29.382375 16807 raft_consensus.cc:374] T b0c0666e4e774091b0edd8ae5af8ef09 P 9eda5e08e6434319b8b3626b9fabf245 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:09:29.382419 16807 raft_consensus.cc:729] T b0c0666e4e774091b0edd8ae5af8ef09 P 9eda5e08e6434319b8b3626b9fabf245 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9eda5e08e6434319b8b3626b9fabf245, State: Initialized, Role: FOLLOWER
I20260810 04:09:29.382529 16807 consensus_queue.cc:260] T b0c0666e4e774091b0edd8ae5af8ef09 P 9eda5e08e6434319b8b3626b9fabf245 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9eda5e08e6434319b8b3626b9fabf245" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 44889 } } peers { permanent_uuid: "7b0dea1b85284b918eb1601d887ae01b" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 46861 } } peers { permanent_uuid: "1fa692fb508741e1be5992f3fd536b48" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 45549 } }
I20260810 04:09:29.382784 16807 ts_tablet_manager.cc:1428] T b0c0666e4e774091b0edd8ae5af8ef09 P 9eda5e08e6434319b8b3626b9fabf245: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260810 04:09:29.382822 16792 heartbeater.cc:503] Master 127.31.32.62:39943 was elected leader, sending a full tablet report...
I20260810 04:09:29.383970 16810 raft_consensus.cc:348] T b0c0666e4e774091b0edd8ae5af8ef09 P 7b0dea1b85284b918eb1601d887ae01b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9eda5e08e6434319b8b3626b9fabf245" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 44889 } } peers { permanent_uuid: "7b0dea1b85284b918eb1601d887ae01b" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 46861 } } peers { permanent_uuid: "1fa692fb508741e1be5992f3fd536b48" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 45549 } }
I20260810 04:09:29.384093 16810 raft_consensus.cc:374] T b0c0666e4e774091b0edd8ae5af8ef09 P 7b0dea1b85284b918eb1601d887ae01b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:09:29.384114 16810 raft_consensus.cc:729] T b0c0666e4e774091b0edd8ae5af8ef09 P 7b0dea1b85284b918eb1601d887ae01b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b0dea1b85284b918eb1601d887ae01b, State: Initialized, Role: FOLLOWER
I20260810 04:09:29.384264 16810 consensus_queue.cc:260] T b0c0666e4e774091b0edd8ae5af8ef09 P 7b0dea1b85284b918eb1601d887ae01b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9eda5e08e6434319b8b3626b9fabf245" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 44889 } } peers { permanent_uuid: "7b0dea1b85284b918eb1601d887ae01b" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 46861 } } peers { permanent_uuid: "1fa692fb508741e1be5992f3fd536b48" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 45549 } }
I20260810 04:09:29.384519 16810 ts_tablet_manager.cc:1428] T b0c0666e4e774091b0edd8ae5af8ef09 P 7b0dea1b85284b918eb1601d887ae01b: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260810 04:09:29.384536 16497 heartbeater.cc:503] Master 127.31.32.62:39943 was elected leader, sending a full tablet report...
W20260810 04:09:29.437953 16646 tablet.cc:2369] T b0c0666e4e774091b0edd8ae5af8ef09 P 1fa692fb508741e1be5992f3fd536b48: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 04:09:29.532907 16498 tablet.cc:2369] T b0c0666e4e774091b0edd8ae5af8ef09 P 7b0dea1b85284b918eb1601d887ae01b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 04:09:29.589979 16793 tablet.cc:2369] T b0c0666e4e774091b0edd8ae5af8ef09 P 9eda5e08e6434319b8b3626b9fabf245: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 04:09:29.700227 16811 raft_consensus.cc:482] T b0c0666e4e774091b0edd8ae5af8ef09 P 1fa692fb508741e1be5992f3fd536b48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 04:09:29.700362 16811 raft_consensus.cc:504] T b0c0666e4e774091b0edd8ae5af8ef09 P 1fa692fb508741e1be5992f3fd536b48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9eda5e08e6434319b8b3626b9fabf245" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 44889 } } peers { permanent_uuid: "7b0dea1b85284b918eb1601d887ae01b" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 46861 } } peers { permanent_uuid: "1fa692fb508741e1be5992f3fd536b48" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 45549 } }
I20260810 04:09:29.700793 16811 leader_election.cc:288] T b0c0666e4e774091b0edd8ae5af8ef09 P 1fa692fb508741e1be5992f3fd536b48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9eda5e08e6434319b8b3626b9fabf245 (127.31.32.3:44889), 7b0dea1b85284b918eb1601d887ae01b (127.31.32.1:46861)
I20260810 04:09:29.704993 16802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:29.701188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.2:42617 (local address 127.31.32.3:44889)
0810 04:09:29.701232 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0810 04:09:29.701233 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 04:09:29.701410 (+ 177us) server_negotiation.cc:407] Connection header received
0810 04:09:29.701450 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:09:29.701451 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:09:29.701464 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:09:29.701545 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:09:29.701765 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:09:29.702518 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:09:29.704189 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:09:29.704601 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:09:29.704692 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:09:29.704704 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0810 04:09:29.704876 (+ 172us) server_negotiation.cc:299] Negotiation successful
0810 04:09:29.704924 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260810 04:09:29.704993 16804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:29.701150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.2:57071 (local address 127.31.32.1:46861)
0810 04:09:29.701201 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0810 04:09:29.701203 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:09:29.701428 (+ 225us) server_negotiation.cc:407] Connection header received
0810 04:09:29.701457 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:09:29.701458 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:09:29.701471 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:09:29.701545 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:09:29.701766 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:09:29.702533 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:09:29.704239 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:09:29.704605 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:09:29.704700 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:09:29.704712 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0810 04:09:29.704873 (+ 161us) server_negotiation.cc:299] Negotiation successful
0810 04:09:29.704918 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260810 04:09:29.705017 16816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:29.701055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.3:44889 (local address 127.31.32.2:42617)
0810 04:09:29.701227 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0810 04:09:29.701236 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 04:09:29.701289 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:09:29.701572 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:09:29.701572 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:09:29.701574 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:09:29.701678 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:09:29.701681 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:09:29.702619 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:09:29.702620 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:09:29.704089 (+ 1469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:09:29.704091 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:09:29.704695 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:09:29.704696 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:09:29.704780 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:09:29.704784 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 04:09:29.704866 (+ 82us) client_negotiation.cc:240] Negotiation successful
0810 04:09:29.704919 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"mutex_wait_us":80,"spinlock_wait_cycles":15232,"thread_start_us":67,"threads_started":1}
I20260810 04:09:29.705017 16815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:29.701055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.1:46861 (local address 127.31.32.2:57071)
0810 04:09:29.701196 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0810 04:09:29.701208 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 04:09:29.701285 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:09:29.701571 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:09:29.701572 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:09:29.701576 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:09:29.701678 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:09:29.701682 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:09:29.702620 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:09:29.702620 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:09:29.704069 (+ 1449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:09:29.704073 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:09:29.704705 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:09:29.704706 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:09:29.704789 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:09:29.704792 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 04:09:29.704864 (+ 72us) client_negotiation.cc:240] Negotiation successful
0810 04:09:29.704912 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"mutex_wait_us":136,"thread_start_us":71,"threads_started":1}
I20260810 04:09:29.705380 16739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0c0666e4e774091b0edd8ae5af8ef09" candidate_uuid: "1fa692fb508741e1be5992f3fd536b48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9eda5e08e6434319b8b3626b9fabf245" is_pre_election: true
I20260810 04:09:29.705561 16739 raft_consensus.cc:2454] T b0c0666e4e774091b0edd8ae5af8ef09 P 9eda5e08e6434319b8b3626b9fabf245 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fa692fb508741e1be5992f3fd536b48 in term 0.
I20260810 04:09:29.705704 16447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0c0666e4e774091b0edd8ae5af8ef09" candidate_uuid: "1fa692fb508741e1be5992f3fd536b48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b0dea1b85284b918eb1601d887ae01b" is_pre_election: true
I20260810 04:09:29.705816 16519 leader_election.cc:302] T b0c0666e4e774091b0edd8ae5af8ef09 P 1fa692fb508741e1be5992f3fd536b48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fa692fb508741e1be5992f3fd536b48, 9eda5e08e6434319b8b3626b9fabf245; no voters:
I20260810 04:09:29.705871 16447 raft_consensus.cc:2454] T b0c0666e4e774091b0edd8ae5af8ef09 P 7b0dea1b85284b918eb1601d887ae01b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fa692fb508741e1be5992f3fd536b48 in term 0.
I20260810 04:09:29.705938 16811 raft_consensus.cc:2781] T b0c0666e4e774091b0edd8ae5af8ef09 P 1fa692fb508741e1be5992f3fd536b48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 04:09:29.705992 16811 raft_consensus.cc:482] T b0c0666e4e774091b0edd8ae5af8ef09 P 1fa692fb508741e1be5992f3fd536b48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 04:09:29.706008 16811 raft_consensus.cc:3037] T b0c0666e4e774091b0edd8ae5af8ef09 P 1fa692fb508741e1be5992f3fd536b48 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:09:29.706687 16811 raft_consensus.cc:504] T b0c0666e4e774091b0edd8ae5af8ef09 P 1fa692fb508741e1be5992f3fd536b48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9eda5e08e6434319b8b3626b9fabf245" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 44889 } } peers { permanent_uuid: "7b0dea1b85284b918eb1601d887ae01b" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 46861 } } peers { permanent_uuid: "1fa692fb508741e1be5992f3fd536b48" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 45549 } }
I20260810 04:09:29.706840 16811 leader_election.cc:288] T b0c0666e4e774091b0edd8ae5af8ef09 P 1fa692fb508741e1be5992f3fd536b48 [CANDIDATE]: Term 1 election: Requested vote from peers 9eda5e08e6434319b8b3626b9fabf245 (127.31.32.3:44889), 7b0dea1b85284b918eb1601d887ae01b (127.31.32.1:46861)
I20260810 04:09:29.706997 16739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0c0666e4e774091b0edd8ae5af8ef09" candidate_uuid: "1fa692fb508741e1be5992f3fd536b48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9eda5e08e6434319b8b3626b9fabf245"
I20260810 04:09:29.707067 16447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0c0666e4e774091b0edd8ae5af8ef09" candidate_uuid: "1fa692fb508741e1be5992f3fd536b48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b0dea1b85284b918eb1601d887ae01b"
I20260810 04:09:29.707088 16739 raft_consensus.cc:3037] T b0c0666e4e774091b0edd8ae5af8ef09 P 9eda5e08e6434319b8b3626b9fabf245 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:09:29.707154 16447 raft_consensus.cc:3037] T b0c0666e4e774091b0edd8ae5af8ef09 P 7b0dea1b85284b918eb1601d887ae01b [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:09:29.707861 16739 raft_consensus.cc:2454] T b0c0666e4e774091b0edd8ae5af8ef09 P 9eda5e08e6434319b8b3626b9fabf245 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1fa692fb508741e1be5992f3fd536b48 in term 1.
I20260810 04:09:29.707921 16447 raft_consensus.cc:2454] T b0c0666e4e774091b0edd8ae5af8ef09 P 7b0dea1b85284b918eb1601d887ae01b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1fa692fb508741e1be5992f3fd536b48 in term 1.
I20260810 04:09:29.708088 16519 leader_election.cc:302] T b0c0666e4e774091b0edd8ae5af8ef09 P 1fa692fb508741e1be5992f3fd536b48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fa692fb508741e1be5992f3fd536b48, 9eda5e08e6434319b8b3626b9fabf245; no voters:
I20260810 04:09:29.708211 16811 raft_consensus.cc:2781] T b0c0666e4e774091b0edd8ae5af8ef09 P 1fa692fb508741e1be5992f3fd536b48 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:09:29.708434 16811 raft_consensus.cc:686] T b0c0666e4e774091b0edd8ae5af8ef09 P 1fa692fb508741e1be5992f3fd536b48 [term 1 LEADER]: Becoming Leader. State: Replica: 1fa692fb508741e1be5992f3fd536b48, State: Running, Role: LEADER
I20260810 04:09:29.708535 16811 consensus_queue.cc:237] T b0c0666e4e774091b0edd8ae5af8ef09 P 1fa692fb508741e1be5992f3fd536b48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9eda5e08e6434319b8b3626b9fabf245" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 44889 } } peers { permanent_uuid: "7b0dea1b85284b918eb1601d887ae01b" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 46861 } } peers { permanent_uuid: "1fa692fb508741e1be5992f3fd536b48" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 45549 } }
I20260810 04:09:29.709414 16291 catalog_manager.cc:5495] T b0c0666e4e774091b0edd8ae5af8ef09 P 1fa692fb508741e1be5992f3fd536b48 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1fa692fb508741e1be5992f3fd536b48 (127.31.32.2). New cstate: current_term: 1 leader_uuid: "1fa692fb508741e1be5992f3fd536b48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9eda5e08e6434319b8b3626b9fabf245" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 44889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b0dea1b85284b918eb1601d887ae01b" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 46861 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1fa692fb508741e1be5992f3fd536b48" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 45549 } health_report { overall_health: HEALTHY } } }
I20260810 04:09:29.723793 16805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:29.721621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45038 (local address 127.31.32.2:45549)
0810 04:09:29.721657 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0810 04:09:29.721658 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 04:09:29.721670 (+ 12us) server_negotiation.cc:407] Connection header received
0810 04:09:29.721743 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:09:29.721744 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:09:29.721759 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:09:29.721826 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 04:09:29.721940 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:09:29.722446 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:09:29.723168 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:09:29.723345 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:09:29.723396 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:09:29.723533 (+ 137us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 04:09:29.723621 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 04:09:29.723680 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0810 04:09:29.723722 (+ 42us) server_negotiation.cc:299] Negotiation successful
0810 04:09:29.723740 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260810 04:09:29.725459 16447 raft_consensus.cc:1261] T b0c0666e4e774091b0edd8ae5af8ef09 P 7b0dea1b85284b918eb1601d887ae01b [term 1 FOLLOWER]: Refusing update from remote peer 1fa692fb508741e1be5992f3fd536b48: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:09:29.725473 16739 raft_consensus.cc:1261] T b0c0666e4e774091b0edd8ae5af8ef09 P 9eda5e08e6434319b8b3626b9fabf245 [term 1 FOLLOWER]: Refusing update from remote peer 1fa692fb508741e1be5992f3fd536b48: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:09:29.725809 16811 consensus_queue.cc:1035] T b0c0666e4e774091b0edd8ae5af8ef09 P 1fa692fb508741e1be5992f3fd536b48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9eda5e08e6434319b8b3626b9fabf245" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 44889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:09:29.725955 16817 consensus_queue.cc:1035] T b0c0666e4e774091b0edd8ae5af8ef09 P 1fa692fb508741e1be5992f3fd536b48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b0dea1b85284b918eb1601d887ae01b" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 46861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:09:29.728318 16822 mvcc.cc:205] Tried to move back new op lower bound from 7316828035993088000 to 7316828035926896640. Current Snapshot: MvccSnapshot[applied={T|T < 7316828035993088000}]
I20260810 04:09:29.731204 16291 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35912:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 04:09:29.731307 16291 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35912:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 04:09:29.732100 16291 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 39ad49896b2d4abf8bf3e221466c2cf5: Sending DeleteTablet for 3 replicas of tablet b0c0666e4e774091b0edd8ae5af8ef09
I20260810 04:09:29.732491 16718 tablet_service.cc:1514] Processing DeleteTablet for tablet b0c0666e4e774091b0edd8ae5af8ef09 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:09:29 UTC) from {username='kudu', principal='kudu/127.31.32.62@KRBTEST.COM'} at 127.0.0.1:43686
I20260810 04:09:29.732510 16423 tablet_service.cc:1514] Processing DeleteTablet for tablet b0c0666e4e774091b0edd8ae5af8ef09 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:09:29 UTC) from {username='kudu', principal='kudu/127.31.32.62@KRBTEST.COM'} at 127.0.0.1:52852
I20260810 04:09:29.732788 16570 tablet_service.cc:1514] Processing DeleteTablet for tablet b0c0666e4e774091b0edd8ae5af8ef09 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:09:29 UTC) from {username='kudu', principal='kudu/127.31.32.62@KRBTEST.COM'} at 127.0.0.1:45036
I20260810 04:09:29.732877 31872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskws89Io/build/release/bin/kudu with pid 16358
I20260810 04:09:29.732885 16831 tablet_replica.cc:330] T b0c0666e4e774091b0edd8ae5af8ef09 P 9eda5e08e6434319b8b3626b9fabf245: stopping tablet replica
I20260810 04:09:29.732990 16831 raft_consensus.cc:2229] T b0c0666e4e774091b0edd8ae5af8ef09 P 9eda5e08e6434319b8b3626b9fabf245 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 04:09:29.733088 16831 raft_consensus.cc:2258] T b0c0666e4e774091b0edd8ae5af8ef09 P 9eda5e08e6434319b8b3626b9fabf245 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:09:29.736500 16831 ts_tablet_manager.cc:1905] T b0c0666e4e774091b0edd8ae5af8ef09 P 9eda5e08e6434319b8b3626b9fabf245: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:09:29.738559 16831 ts_tablet_manager.cc:1918] T b0c0666e4e774091b0edd8ae5af8ef09 P 9eda5e08e6434319b8b3626b9fabf245: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 04:09:29.738643 16831 log.cc:1196] T b0c0666e4e774091b0edd8ae5af8ef09 P 9eda5e08e6434319b8b3626b9fabf245: Deleting WAL directory at /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/ts-2/wal/wals/b0c0666e4e774091b0edd8ae5af8ef09
I20260810 04:09:29.738945 16831 ts_tablet_manager.cc:1939] T b0c0666e4e774091b0edd8ae5af8ef09 P 9eda5e08e6434319b8b3626b9fabf245: Deleting consensus metadata
I20260810 04:09:29.739421 16273 catalog_manager.cc:4841] TS 9eda5e08e6434319b8b3626b9fabf245 (127.31.32.3:44889): tablet b0c0666e4e774091b0edd8ae5af8ef09 (table test-table [id=5e452ca7c268412c86e30c97968f5f94]) successfully deleted
I20260810 04:09:29.742784 16832 tablet_replica.cc:330] T b0c0666e4e774091b0edd8ae5af8ef09 P 1fa692fb508741e1be5992f3fd536b48: stopping tablet replica
W20260810 04:09:29.742802 16275 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.32.1:46861 (error 108)
I20260810 04:09:29.742916 16832 raft_consensus.cc:2229] T b0c0666e4e774091b0edd8ae5af8ef09 P 1fa692fb508741e1be5992f3fd536b48 [term 1 LEADER]: Raft consensus shutting down.
I20260810 04:09:29.743088 16832 raft_consensus.cc:2258] T b0c0666e4e774091b0edd8ae5af8ef09 P 1fa692fb508741e1be5992f3fd536b48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:09:29.743371 16800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:29.743221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.1:46861 (local address 127.0.0.1:52854)
0810 04:09:29.743289 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0810 04:09:29.743335 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.1:46861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260810 04:09:29.743394 31872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskws89Io/build/release/bin/kudu with pid 16503
W20260810 04:09:29.744529 16275 catalog_manager.cc:4584] TS 7b0dea1b85284b918eb1601d887ae01b (127.31.32.1:46861): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b0c0666e4e774091b0edd8ae5af8ef09: Network error: Client connection negotiation failed: client connection to 127.31.32.1:46861: connect: Connection refused (error 111)
I20260810 04:09:29.751562 31872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskws89Io/build/release/bin/kudu with pid 16650
I20260810 04:09:29.751757 16800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:29.751593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.2:45549 (local address 127.0.0.1:45046)
0810 04:09:29.751675 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0810 04:09:29.751719 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.2:45549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260810 04:09:29.764401 31872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskws89Io/build/release/bin/kudu with pid 16257
I20260810 04:09:49.771767 31872 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskws89Io/build/release/bin/kudu
/tmp/dist-test-taskws89Io/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.32.62:39943
--webserver_interface=127.31.32.62
--webserver_port=42757
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.62.keytab
--principal=kudu/127.31.32.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.32.20:40835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.32.62:39943
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:09:49.855687 16834 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:09:49.855888 16834 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:09:49.855911 16834 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:09:49.857165 16834 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260810 04:09:49.857206 16834 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260810 04:09:49.857219 16834 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260810 04:09:49.857231 16834 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:09:49.857276 16834 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:09:49.857302 16834 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:09:49.857326 16834 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:09:49.857347 16834 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:09:49.858875 16834 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.32.20:40835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.31.32.62:39943
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.62.keytab
--principal=kudu/127.31.32.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.32.62:39943
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.32.62
--webserver_port=42757
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16834
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:09:49.859099 16834 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:09:49.859339 16834 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:09:49.868762 16841 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:09:49.868762 16840 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:09:49.868798 16834 server_base.cc:1034] running on GCE node
W20260810 04:09:49.868758 16843 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:09:49.869225 16834 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:09:49.869555 16834 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:09:49.870693 16834 hybrid_clock.cc:648] HybridClock initialized: now 1786334989870695 us; error 39 us; skew 500 ppm
Aug 10 04:09:49 dist-test-slave-8w9v krb5kdc[16241](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334989, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:09:49.877305 16834 init.cc:377] Logged in from keytab as kudu/127.31.32.62@KRBTEST.COM (short username kudu)
I20260810 04:09:49.877801 16834 webserver.cc:460] Webserver started at http://127.31.32.62:42757/ using document root <none> and password file <none>
I20260810 04:09:49.878057 16834 fs_manager.cc:359] Metadata directory not provided
I20260810 04:09:49.878144 16834 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:09:49.879549 16834 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 04:09:49.880280 16852 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:09:49.880455 16834 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:09:49.880556 16834 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/master-0/data,/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/master-0/wal
uuid: "39ad49896b2d4abf8bf3e221466c2cf5"
format_stamp: "Formatted at 2026-08-10 04:09:28 on dist-test-slave-8w9v"
server_key: "4465989c929bab8a58259d84eea563fe"
server_key_iv: "8cc3a42036dbb78e710b5de52047c8df"
server_key_version: "encryptionkey@0"
I20260810 04:09:49.880970 16834 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:09:49.897147 16834 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:09:49.899165 16834 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:09:49.899327 16834 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:09:49.903499 16834 rpc_server.cc:307] RPC server started. Bound to: 127.31.32.62:39943
I20260810 04:09:49.903537 16910 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.32.62:39943 every 8 connection(s)
I20260810 04:09:49.903923 16834 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/master-0/data/info.pb
I20260810 04:09:49.905467 16911 sys_catalog.cc:263] Verifying existing consensus state
I20260810 04:09:49.906066 16911 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39ad49896b2d4abf8bf3e221466c2cf5: Bootstrap starting.
I20260810 04:09:49.908174 31872 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskws89Io/build/release/bin/kudu as pid 16834
I20260810 04:09:49.908522 31872 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskws89Io/build/release/bin/kudu
/tmp/dist-test-taskws89Io/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.32.1:46861
--local_ip_for_outbound_sockets=127.31.32.1
--tserver_master_addrs=127.31.32.62:39943
--webserver_port=45745
--webserver_interface=127.31.32.1
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.1.keytab
--principal=kudu/127.31.32.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.32.20:40835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 04:09:49.914602 16911 log.cc:824] T 00000000000000000000000000000000 P 39ad49896b2d4abf8bf3e221466c2cf5: Log is configured to *not* fsync() on all Append() calls
I20260810 04:09:49.916610 16911 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39ad49896b2d4abf8bf3e221466c2cf5: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 04:09:49.916936 16911 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39ad49896b2d4abf8bf3e221466c2cf5: Bootstrap complete.
I20260810 04:09:49.919232 16911 raft_consensus.cc:348] T 00000000000000000000000000000000 P 39ad49896b2d4abf8bf3e221466c2cf5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39ad49896b2d4abf8bf3e221466c2cf5" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 39943 } }
I20260810 04:09:49.919461 16911 raft_consensus.cc:729] T 00000000000000000000000000000000 P 39ad49896b2d4abf8bf3e221466c2cf5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39ad49896b2d4abf8bf3e221466c2cf5, State: Initialized, Role: FOLLOWER
I20260810 04:09:49.919628 16911 consensus_queue.cc:260] T 00000000000000000000000000000000 P 39ad49896b2d4abf8bf3e221466c2cf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39ad49896b2d4abf8bf3e221466c2cf5" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 39943 } }
I20260810 04:09:49.919690 16911 raft_consensus.cc:388] T 00000000000000000000000000000000 P 39ad49896b2d4abf8bf3e221466c2cf5 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 04:09:49.919745 16911 raft_consensus.cc:482] T 00000000000000000000000000000000 P 39ad49896b2d4abf8bf3e221466c2cf5 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 04:09:49.919793 16911 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 39ad49896b2d4abf8bf3e221466c2cf5 [term 1 FOLLOWER]: Advancing to term 2
I20260810 04:09:49.920862 16911 raft_consensus.cc:504] T 00000000000000000000000000000000 P 39ad49896b2d4abf8bf3e221466c2cf5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39ad49896b2d4abf8bf3e221466c2cf5" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 39943 } }
I20260810 04:09:49.920976 16911 leader_election.cc:302] T 00000000000000000000000000000000 P 39ad49896b2d4abf8bf3e221466c2cf5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 39ad49896b2d4abf8bf3e221466c2cf5; no voters:
I20260810 04:09:49.921185 16911 leader_election.cc:288] T 00000000000000000000000000000000 P 39ad49896b2d4abf8bf3e221466c2cf5 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260810 04:09:49.921226 16915 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 39ad49896b2d4abf8bf3e221466c2cf5 [term 2 FOLLOWER]: Leader election won for term 2
I20260810 04:09:49.921396 16915 raft_consensus.cc:686] T 00000000000000000000000000000000 P 39ad49896b2d4abf8bf3e221466c2cf5 [term 2 LEADER]: Becoming Leader. State: Replica: 39ad49896b2d4abf8bf3e221466c2cf5, State: Running, Role: LEADER
I20260810 04:09:49.921474 16911 sys_catalog.cc:564] T 00000000000000000000000000000000 P 39ad49896b2d4abf8bf3e221466c2cf5 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:09:49.921541 16915 consensus_queue.cc:237] T 00000000000000000000000000000000 P 39ad49896b2d4abf8bf3e221466c2cf5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39ad49896b2d4abf8bf3e221466c2cf5" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 39943 } }
I20260810 04:09:49.921952 16916 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39ad49896b2d4abf8bf3e221466c2cf5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "39ad49896b2d4abf8bf3e221466c2cf5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39ad49896b2d4abf8bf3e221466c2cf5" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 39943 } } }
I20260810 04:09:49.922039 16916 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39ad49896b2d4abf8bf3e221466c2cf5 [sys.catalog]: This master's current role is: LEADER
I20260810 04:09:49.922168 16915 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39ad49896b2d4abf8bf3e221466c2cf5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 39ad49896b2d4abf8bf3e221466c2cf5. Latest consensus state: current_term: 2 leader_uuid: "39ad49896b2d4abf8bf3e221466c2cf5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39ad49896b2d4abf8bf3e221466c2cf5" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 39943 } } }
I20260810 04:09:49.922382 16915 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39ad49896b2d4abf8bf3e221466c2cf5 [sys.catalog]: This master's current role is: LEADER
I20260810 04:09:49.922739 16925 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 04:09:49.923560 16925 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 04:09:49.923659 16925 catalog_manager.cc:1259] Loaded cluster ID: 0bedc7d1051749e099934262464f9bde
I20260810 04:09:49.923691 16925 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 04:09:49.924980 16925 catalog_manager.cc:1504] Loading token signing keys...
I20260810 04:09:49.925194 16925 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 39ad49896b2d4abf8bf3e221466c2cf5: Loaded TSK: 0
I20260810 04:09:49.937417 16925 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 39ad49896b2d4abf8bf3e221466c2cf5: Generated new TSK 1
I20260810 04:09:49.937506 16925 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 04:09:49.996716 16913 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:09:49.996903 16913 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:09:49.996927 16913 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:09:49.998023 16913 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:09:49.998062 16913 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:09:49.998087 16913 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.32.1
I20260810 04:09:49.999668 16913 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.32.20:40835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.1.keytab
--principal=kudu/127.31.32.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.32.1:46861
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.32.1
--webserver_port=45745
--tserver_master_addrs=127.31.32.62:39943
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16913
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.32.1
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:09:49.999887 16913 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:09:50.000185 16913 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:09:50.006137 16940 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:09:50.006120 16941 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:09:50.006124 16943 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:09:50.006433 16913 server_base.cc:1034] running on GCE node
I20260810 04:09:50.006670 16913 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:09:50.006973 16913 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:09:50.008191 16913 hybrid_clock.cc:648] HybridClock initialized: now 1786334990008168 us; error 40 us; skew 500 ppm
Aug 10 04:09:50 dist-test-slave-8w9v krb5kdc[16241](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334990, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:09:50.010974 16913 init.cc:377] Logged in from keytab as kudu/127.31.32.1@KRBTEST.COM (short username kudu)
I20260810 04:09:50.011487 16913 webserver.cc:460] Webserver started at http://127.31.32.1:45745/ using document root <none> and password file <none>
I20260810 04:09:50.011735 16913 fs_manager.cc:359] Metadata directory not provided
I20260810 04:09:50.011807 16913 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:09:50.013619 16913 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:09:50.014400 16950 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:09:50.014602 16913 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:09:50.014703 16913 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/ts-0/data,/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/ts-0/wal
uuid: "7b0dea1b85284b918eb1601d887ae01b"
format_stamp: "Formatted at 2026-08-10 04:09:28 on dist-test-slave-8w9v"
server_key: "b5b53b670d118fb11fea1357903cd359"
server_key_iv: "a2b1e2ab3dd73110d77044d83054bdc6"
server_key_version: "encryptionkey@0"
I20260810 04:09:50.015151 16913 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:09:50.032256 16913 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:09:50.034694 16913 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:09:50.034865 16913 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:09:50.035130 16913 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:09:50.035638 16957 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260810 04:09:50.037333 16913 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260810 04:09:50.037395 16913 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260810 04:09:50.037429 16913 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260810 04:09:50.038277 16913 ts_tablet_manager.cc:610] Registered 1 tablets
I20260810 04:09:50.038324 16913 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260810 04:09:50.038430 16957 tablet_bootstrap.cc:492] T b0c0666e4e774091b0edd8ae5af8ef09 P 7b0dea1b85284b918eb1601d887ae01b: Bootstrap starting.
I20260810 04:09:50.044865 16913 rpc_server.cc:307] RPC server started. Bound to: 127.31.32.1:46861
I20260810 04:09:50.045136 17074 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.32.1:46861 every 8 connection(s)
I20260810 04:09:50.045266 16913 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/ts-0/data/info.pb
I20260810 04:09:50.050203 16957 log.cc:824] T b0c0666e4e774091b0edd8ae5af8ef09 P 7b0dea1b85284b918eb1601d887ae01b: Log is configured to *not* fsync() on all Append() calls
I20260810 04:09:50.052968 16957 tablet_bootstrap.cc:492] T b0c0666e4e774091b0edd8ae5af8ef09 P 7b0dea1b85284b918eb1601d887ae01b: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 04:09:50.053398 16957 tablet_bootstrap.cc:492] T b0c0666e4e774091b0edd8ae5af8ef09 P 7b0dea1b85284b918eb1601d887ae01b: Bootstrap complete.
I20260810 04:09:50.054335 31872 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskws89Io/build/release/bin/kudu as pid 16913
Aug 10 04:09:50 dist-test-slave-8w9v krb5kdc[16241](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334990, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.1@KRBTEST.COM for kudu/127.31.32.62@KRBTEST.COM
I20260810 04:09:50.054684 31872 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskws89Io/build/release/bin/kudu
/tmp/dist-test-taskws89Io/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.32.2:45549
--local_ip_for_outbound_sockets=127.31.32.2
--tserver_master_addrs=127.31.32.62:39943
--webserver_port=39677
--webserver_interface=127.31.32.2
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.2.keytab
--principal=kudu/127.31.32.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.32.20:40835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 04:09:50.054898 16957 ts_tablet_manager.cc:1397] T b0c0666e4e774091b0edd8ae5af8ef09 P 7b0dea1b85284b918eb1601d887ae01b: Time spent bootstrapping tablet: real 0.017s user 0.007s sys 0.009s
I20260810 04:09:50.057471 16957 raft_consensus.cc:348] T b0c0666e4e774091b0edd8ae5af8ef09 P 7b0dea1b85284b918eb1601d887ae01b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9eda5e08e6434319b8b3626b9fabf245" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 44889 } } peers { permanent_uuid: "7b0dea1b85284b918eb1601d887ae01b" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 46861 } } peers { permanent_uuid: "1fa692fb508741e1be5992f3fd536b48" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 45549 } }
I20260810 04:09:50.057885 16957 raft_consensus.cc:729] T b0c0666e4e774091b0edd8ae5af8ef09 P 7b0dea1b85284b918eb1601d887ae01b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b0dea1b85284b918eb1601d887ae01b, State: Initialized, Role: FOLLOWER
I20260810 04:09:50.058060 16957 consensus_queue.cc:260] T b0c0666e4e774091b0edd8ae5af8ef09 P 7b0dea1b85284b918eb1601d887ae01b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9eda5e08e6434319b8b3626b9fabf245" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 44889 } } peers { permanent_uuid: "7b0dea1b85284b918eb1601d887ae01b" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 46861 } } peers { permanent_uuid: "1fa692fb508741e1be5992f3fd536b48" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 45549 } }
I20260810 04:09:50.058349 16957 ts_tablet_manager.cc:1428] T b0c0666e4e774091b0edd8ae5af8ef09 P 7b0dea1b85284b918eb1601d887ae01b: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260810 04:09:50.061570 17080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:50.046138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.1:60979 (local address 127.31.32.62:39943)
0810 04:09:50.046333 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0810 04:09:50.046342 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 04:09:50.046381 (+ 39us) server_negotiation.cc:407] Connection header received
0810 04:09:50.047339 (+ 958us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:09:50.047347 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:09:50.047405 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:09:50.047544 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:09:50.048633 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:09:50.050035 (+ 1402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:09:50.051578 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:09:50.052327 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:09:50.052441 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:09:50.057081 (+ 4640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:09:50.057085 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:09:50.057085 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:09:50.057097 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:09:50.059360 (+ 2263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:09:50.059914 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:09:50.059917 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:09:50.059917 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:09:50.060025 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:09:50.060453 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:09:50.060454 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:09:50.060454 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:09:50.060718 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:09:50.060804 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0810 04:09:50.061232 (+ 428us) server_negotiation.cc:299] Negotiation successful
0810 04:09:50.061367 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":105,"threads_started":1}
I20260810 04:09:50.061632 17079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:50.045986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:39943 (local address 127.31.32.1:60979)
0810 04:09:50.046158 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0810 04:09:50.046217 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0810 04:09:50.047056 (+ 839us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:09:50.047627 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:09:50.047633 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:09:50.048140 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:09:50.048481 (+ 341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:09:50.048489 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:09:50.050210 (+ 1721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:09:50.050212 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:09:50.051475 (+ 1263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:09:50.051479 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:09:50.052442 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:09:50.052444 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:09:50.052605 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:09:50.053234 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:09:50.053244 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:09:50.056971 (+ 3727us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:09:50.059453 (+ 2482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:09:50.059461 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:09:50.059463 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:09:50.059818 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:09:50.060111 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:09:50.060112 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:09:50.060113 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:09:50.060303 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:09:50.060809 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:09:50.060817 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:09:50.061007 (+ 190us) client_negotiation.cc:769] Sending connection context
0810 04:09:50.061135 (+ 128us) client_negotiation.cc:240] Negotiation successful
0810 04:09:50.061354 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":86,"threads_started":1}
I20260810 04:09:50.062903 17076 heartbeater.cc:347] Connected to a master server at 127.31.32.62:39943
I20260810 04:09:50.063011 17076 heartbeater.cc:464] Registering TS with master...
I20260810 04:09:50.063194 17076 heartbeater.cc:511] Master 127.31.32.62:39943 requested a full tablet report, sending...
I20260810 04:09:50.063920 16872 ts_manager.cc:194] Registered new tserver with Master: 7b0dea1b85284b918eb1601d887ae01b (127.31.32.1:46861)
I20260810 04:09:50.065001 16872 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.32.1@KRBTEST.COM'} at 127.31.32.1:60979
I20260810 04:09:50.065837 17076 heartbeater.cc:503] Master 127.31.32.62:39943 was elected leader, sending a full tablet report...
Aug 10 04:09:50 dist-test-slave-8w9v krb5kdc[16241](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334989, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.62@KRBTEST.COM for kudu/127.31.32.1@KRBTEST.COM
I20260810 04:09:50.072798 17084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:50.064617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.1:46861 (local address 127.0.0.1:56306)
0810 04:09:50.065006 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0810 04:09:50.065048 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 04:09:50.065134 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:09:50.065441 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:09:50.065445 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:09:50.066237 (+ 792us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:09:50.066326 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:09:50.066329 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:09:50.067119 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:09:50.067120 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:09:50.067680 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:09:50.067682 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:09:50.068161 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:09:50.068162 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:09:50.068246 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:09:50.068692 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:09:50.068704 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:09:50.069931 (+ 1227us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:09:50.071626 (+ 1695us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:09:50.071632 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:09:50.071634 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:09:50.071832 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:09:50.072050 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:09:50.072051 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:09:50.072051 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:09:50.072114 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:09:50.072423 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:09:50.072429 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:09:50.072481 (+ 52us) client_negotiation.cc:769] Sending connection context
0810 04:09:50.072579 (+ 98us) client_negotiation.cc:240] Negotiation successful
0810 04:09:50.072700 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":84,"threads_started":1}
I20260810 04:09:50.072798 17085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:50.064689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56306 (local address 127.31.32.1:46861)
0810 04:09:50.064837 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0810 04:09:50.064842 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 04:09:50.065230 (+ 388us) server_negotiation.cc:407] Connection header received
0810 04:09:50.065272 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:09:50.065334 (+ 62us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:09:50.065357 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:09:50.065440 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:09:50.066392 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:09:50.067032 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:09:50.067781 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:09:50.068064 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:09:50.068165 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:09:50.070018 (+ 1853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:09:50.070026 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:09:50.070027 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:09:50.070034 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:09:50.071530 (+ 1496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:09:50.071913 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:09:50.071916 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:09:50.071916 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:09:50.071974 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:09:50.072201 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:09:50.072202 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:09:50.072202 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:09:50.072341 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:09:50.072430 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0810 04:09:50.072616 (+ 186us) server_negotiation.cc:299] Negotiation successful
0810 04:09:50.072702 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":82,"threads_started":1}
I20260810 04:09:50.073172 17004 tablet_service.cc:1514] Processing DeleteTablet for tablet b0c0666e4e774091b0edd8ae5af8ef09 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:09:29 UTC) from {username='kudu', principal='kudu/127.31.32.62@KRBTEST.COM'} at 127.0.0.1:56306
I20260810 04:09:50.073675 17086 tablet_replica.cc:330] stopping tablet replica
I20260810 04:09:50.073796 17086 raft_consensus.cc:2229] T b0c0666e4e774091b0edd8ae5af8ef09 P 7b0dea1b85284b918eb1601d887ae01b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 04:09:50.073869 17086 raft_consensus.cc:2258] T b0c0666e4e774091b0edd8ae5af8ef09 P 7b0dea1b85284b918eb1601d887ae01b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:09:50.074203 17086 ts_tablet_manager.cc:1905] T b0c0666e4e774091b0edd8ae5af8ef09 P 7b0dea1b85284b918eb1601d887ae01b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:09:50.077064 17086 ts_tablet_manager.cc:1918] T b0c0666e4e774091b0edd8ae5af8ef09 P 7b0dea1b85284b918eb1601d887ae01b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 04:09:50.077152 17086 log.cc:1196] T b0c0666e4e774091b0edd8ae5af8ef09 P 7b0dea1b85284b918eb1601d887ae01b: Deleting WAL directory at /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/ts-0/wal/wals/b0c0666e4e774091b0edd8ae5af8ef09
I20260810 04:09:50.077445 17086 ts_tablet_manager.cc:1939] T b0c0666e4e774091b0edd8ae5af8ef09 P 7b0dea1b85284b918eb1601d887ae01b: Deleting consensus metadata
I20260810 04:09:50.078011 16854 catalog_manager.cc:4841] TS 7b0dea1b85284b918eb1601d887ae01b (127.31.32.1:46861): tablet b0c0666e4e774091b0edd8ae5af8ef09 (table test-table [id=5e452ca7c268412c86e30c97968f5f94]) successfully deleted
W20260810 04:09:50.145565 17082 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:09:50.145771 17082 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:09:50.145812 17082 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:09:50.146991 17082 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:09:50.147050 17082 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:09:50.147119 17082 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.32.2
I20260810 04:09:50.148545 17082 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.32.20:40835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.2.keytab
--principal=kudu/127.31.32.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.32.2:45549
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.32.2
--webserver_port=39677
--tserver_master_addrs=127.31.32.62:39943
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17082
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.32.2
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:09:50.148800 17082 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:09:50.149046 17082 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:09:50.153949 17091 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:09:50.153996 17092 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:09:50.154034 17095 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:09:50.154248 17082 server_base.cc:1034] running on GCE node
I20260810 04:09:50.154500 17082 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:09:50.154763 17082 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:09:50.155925 17082 hybrid_clock.cc:648] HybridClock initialized: now 1786334990155911 us; error 25 us; skew 500 ppm
Aug 10 04:09:50 dist-test-slave-8w9v krb5kdc[16241](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334990, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:09:50.158444 17082 init.cc:377] Logged in from keytab as kudu/127.31.32.2@KRBTEST.COM (short username kudu)
I20260810 04:09:50.158838 17082 webserver.cc:460] Webserver started at http://127.31.32.2:39677/ using document root <none> and password file <none>
I20260810 04:09:50.159034 17082 fs_manager.cc:359] Metadata directory not provided
I20260810 04:09:50.159086 17082 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:09:50.160405 17082 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260810 04:09:50.161051 17103 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:09:50.161273 17082 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260810 04:09:50.161387 17082 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/ts-1/data,/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/ts-1/wal
uuid: "1fa692fb508741e1be5992f3fd536b48"
format_stamp: "Formatted at 2026-08-10 04:09:29 on dist-test-slave-8w9v"
server_key: "64a1f9ecbd6b76abc5e22845cee6b701"
server_key_iv: "8518bbb579c2bd18ac4604451e2b4cc5"
server_key_version: "encryptionkey@0"
I20260810 04:09:50.161712 17082 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:09:50.179517 17082 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:09:50.181804 17082 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:09:50.181989 17082 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:09:50.182231 17082 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:09:50.182693 17110 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260810 04:09:50.184373 17082 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260810 04:09:50.184423 17082 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260810 04:09:50.184449 17082 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260810 04:09:50.185135 17082 ts_tablet_manager.cc:610] Registered 1 tablets
I20260810 04:09:50.185168 17082 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260810 04:09:50.185268 17110 tablet_bootstrap.cc:492] T b0c0666e4e774091b0edd8ae5af8ef09 P 1fa692fb508741e1be5992f3fd536b48: Bootstrap starting.
I20260810 04:09:50.186800 17110 log.cc:824] T b0c0666e4e774091b0edd8ae5af8ef09 P 1fa692fb508741e1be5992f3fd536b48: Log is configured to *not* fsync() on all Append() calls
I20260810 04:09:50.190068 17110 tablet_bootstrap.cc:492] T b0c0666e4e774091b0edd8ae5af8ef09 P 1fa692fb508741e1be5992f3fd536b48: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 04:09:50.190412 17110 tablet_bootstrap.cc:492] T b0c0666e4e774091b0edd8ae5af8ef09 P 1fa692fb508741e1be5992f3fd536b48: Bootstrap complete.
I20260810 04:09:50.190569 17110 ts_tablet_manager.cc:1397] T b0c0666e4e774091b0edd8ae5af8ef09 P 1fa692fb508741e1be5992f3fd536b48: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.003s
I20260810 04:09:50.192165 17110 raft_consensus.cc:348] T b0c0666e4e774091b0edd8ae5af8ef09 P 1fa692fb508741e1be5992f3fd536b48 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9eda5e08e6434319b8b3626b9fabf245" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 44889 } } peers { permanent_uuid: "7b0dea1b85284b918eb1601d887ae01b" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 46861 } } peers { permanent_uuid: "1fa692fb508741e1be5992f3fd536b48" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 45549 } }
I20260810 04:09:50.192487 17110 raft_consensus.cc:729] T b0c0666e4e774091b0edd8ae5af8ef09 P 1fa692fb508741e1be5992f3fd536b48 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fa692fb508741e1be5992f3fd536b48, State: Initialized, Role: FOLLOWER
I20260810 04:09:50.192667 17082 rpc_server.cc:307] RPC server started. Bound to: 127.31.32.2:45549
I20260810 04:09:50.192725 17227 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.32.2:45549 every 8 connection(s)
I20260810 04:09:50.192677 17110 consensus_queue.cc:260] T b0c0666e4e774091b0edd8ae5af8ef09 P 1fa692fb508741e1be5992f3fd536b48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9eda5e08e6434319b8b3626b9fabf245" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 44889 } } peers { permanent_uuid: "7b0dea1b85284b918eb1601d887ae01b" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 46861 } } peers { permanent_uuid: "1fa692fb508741e1be5992f3fd536b48" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 45549 } }
I20260810 04:09:50.193051 17082 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/ts-1/data/info.pb
I20260810 04:09:50.193063 17110 ts_tablet_manager.cc:1428] T b0c0666e4e774091b0edd8ae5af8ef09 P 1fa692fb508741e1be5992f3fd536b48: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
Aug 10 04:09:50 dist-test-slave-8w9v krb5kdc[16241](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334990, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.2@KRBTEST.COM for kudu/127.31.32.62@KRBTEST.COM
I20260810 04:09:50.202277 17080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:50.193722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.2:35539 (local address 127.31.32.62:39943)
0810 04:09:50.193767 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0810 04:09:50.193769 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:09:50.193954 (+ 185us) server_negotiation.cc:407] Connection header received
0810 04:09:50.194068 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:09:50.194069 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:09:50.194078 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:09:50.194141 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:09:50.194765 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:09:50.195278 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:09:50.196345 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:09:50.196542 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:09:50.196597 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:09:50.199321 (+ 2724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:09:50.199322 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:09:50.199323 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:09:50.199345 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:09:50.200935 (+ 1590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:09:50.201354 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:09:50.201355 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:09:50.201355 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:09:50.201407 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:09:50.201724 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:09:50.201724 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:09:50.201724 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:09:50.201826 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:09:50.201878 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0810 04:09:50.202121 (+ 243us) server_negotiation.cc:299] Negotiation successful
0810 04:09:50.202192 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260810 04:09:50.202487 17232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:50.193668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:39943 (local address 127.31.32.2:35539)
0810 04:09:50.193822 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0810 04:09:50.193862 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0810 04:09:50.194003 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:09:50.194182 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:09:50.194185 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:09:50.194452 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:09:50.194657 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:09:50.194662 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:09:50.195398 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:09:50.195399 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:09:50.196254 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:09:50.196257 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:09:50.196615 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:09:50.196616 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:09:50.196708 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:09:50.197302 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:09:50.197317 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:09:50.199218 (+ 1901us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:09:50.201026 (+ 1808us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:09:50.201032 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:09:50.201034 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:09:50.201264 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:09:50.201501 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:09:50.201503 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:09:50.201503 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:09:50.201655 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:09:50.201886 (+ 231us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:09:50.201891 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:09:50.202001 (+ 110us) client_negotiation.cc:769] Sending connection context
0810 04:09:50.202093 (+ 92us) client_negotiation.cc:240] Negotiation successful
0810 04:09:50.202211 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":85,"threads_started":1}
I20260810 04:09:50.202476 31872 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskws89Io/build/release/bin/kudu as pid 17082
I20260810 04:09:50.202844 31872 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskws89Io/build/release/bin/kudu
/tmp/dist-test-taskws89Io/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.32.3:44889
--local_ip_for_outbound_sockets=127.31.32.3
--tserver_master_addrs=127.31.32.62:39943
--webserver_port=43127
--webserver_interface=127.31.32.3
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.3.keytab
--principal=kudu/127.31.32.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.32.20:40835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 04:09:50.203022 17229 heartbeater.cc:347] Connected to a master server at 127.31.32.62:39943
I20260810 04:09:50.203119 17229 heartbeater.cc:464] Registering TS with master...
I20260810 04:09:50.203316 17229 heartbeater.cc:511] Master 127.31.32.62:39943 requested a full tablet report, sending...
I20260810 04:09:50.203920 16872 ts_manager.cc:194] Registered new tserver with Master: 1fa692fb508741e1be5992f3fd536b48 (127.31.32.2:45549)
I20260810 04:09:50.204566 16872 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.32.2@KRBTEST.COM'} at 127.31.32.2:35539
Aug 10 04:09:50 dist-test-slave-8w9v krb5kdc[16241](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334989, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.62@KRBTEST.COM for kudu/127.31.32.2@KRBTEST.COM
I20260810 04:09:50.212873 17084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:50.204253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.2:45549 (local address 127.0.0.1:59496)
0810 04:09:50.204308 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0810 04:09:50.204320 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 04:09:50.204366 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:09:50.205098 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:09:50.205099 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:09:50.205273 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:09:50.205324 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:09:50.205326 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:09:50.206617 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:09:50.206618 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:09:50.207158 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:09:50.207160 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:09:50.207866 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:09:50.207867 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:09:50.207917 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:09:50.208409 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:09:50.208414 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:09:50.209592 (+ 1178us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:09:50.211778 (+ 2186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:09:50.211779 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:09:50.211780 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:09:50.211939 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:09:50.212216 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:09:50.212217 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:09:50.212217 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:09:50.212305 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:09:50.212621 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:09:50.212622 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:09:50.212660 (+ 38us) client_negotiation.cc:769] Sending connection context
0810 04:09:50.212718 (+ 58us) client_negotiation.cc:240] Negotiation successful
0810 04:09:50.212810 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260810 04:09:50.212996 17234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:50.204728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59496 (local address 127.31.32.2:45549)
0810 04:09:50.204909 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0810 04:09:50.204915 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 04:09:50.204926 (+ 11us) server_negotiation.cc:407] Connection header received
0810 04:09:50.204963 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:09:50.204965 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:09:50.204990 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:09:50.205417 (+ 427us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:09:50.205472 (+ 55us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:09:50.206526 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:09:50.207246 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:09:50.207786 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:09:50.207868 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:09:50.209676 (+ 1808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:09:50.209684 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:09:50.209685 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:09:50.209692 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:09:50.211685 (+ 1993us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:09:50.212019 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:09:50.212022 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:09:50.212023 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:09:50.212086 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:09:50.212382 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:09:50.212383 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:09:50.212384 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:09:50.212544 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:09:50.212636 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0810 04:09:50.212777 (+ 141us) server_negotiation.cc:299] Negotiation successful
0810 04:09:50.212863 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":92,"threads_started":1}
I20260810 04:09:50.213358 17158 tablet_service.cc:1514] Processing DeleteTablet for tablet b0c0666e4e774091b0edd8ae5af8ef09 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:09:29 UTC) from {username='kudu', principal='kudu/127.31.32.62@KRBTEST.COM'} at 127.0.0.1:59496
I20260810 04:09:50.213969 17235 tablet_replica.cc:330] stopping tablet replica
I20260810 04:09:50.214087 17235 raft_consensus.cc:2229] T b0c0666e4e774091b0edd8ae5af8ef09 P 1fa692fb508741e1be5992f3fd536b48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 04:09:50.214181 17235 raft_consensus.cc:2258] T b0c0666e4e774091b0edd8ae5af8ef09 P 1fa692fb508741e1be5992f3fd536b48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:09:50.214495 17235 ts_tablet_manager.cc:1905] T b0c0666e4e774091b0edd8ae5af8ef09 P 1fa692fb508741e1be5992f3fd536b48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:09:50.216848 17235 ts_tablet_manager.cc:1918] T b0c0666e4e774091b0edd8ae5af8ef09 P 1fa692fb508741e1be5992f3fd536b48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 04:09:50.216912 17235 log.cc:1196] T b0c0666e4e774091b0edd8ae5af8ef09 P 1fa692fb508741e1be5992f3fd536b48: Deleting WAL directory at /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/ts-1/wal/wals/b0c0666e4e774091b0edd8ae5af8ef09
I20260810 04:09:50.217149 17235 ts_tablet_manager.cc:1939] T b0c0666e4e774091b0edd8ae5af8ef09 P 1fa692fb508741e1be5992f3fd536b48: Deleting consensus metadata
I20260810 04:09:50.217710 16853 catalog_manager.cc:4841] TS 1fa692fb508741e1be5992f3fd536b48 (127.31.32.2:45549): tablet b0c0666e4e774091b0edd8ae5af8ef09 (table test-table [id=5e452ca7c268412c86e30c97968f5f94]) successfully deleted
W20260810 04:09:50.286985 17233 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:09:50.287204 17233 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:09:50.287238 17233 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:09:50.288367 17233 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:09:50.288429 17233 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:09:50.288470 17233 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.32.3
I20260810 04:09:50.289929 17233 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.32.20:40835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.3.keytab
--principal=kudu/127.31.32.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.32.3:44889
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.32.3
--webserver_port=43127
--tserver_master_addrs=127.31.32.62:39943
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17233
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.32.3
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:09:50.290189 17233 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:09:50.290412 17233 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:09:50.295336 17244 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:09:50.295348 17241 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:09:50.295468 17233 server_base.cc:1034] running on GCE node
W20260810 04:09:50.295584 17240 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:09:50.295828 17233 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:09:50.296097 17233 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:09:50.297279 17233 hybrid_clock.cc:648] HybridClock initialized: now 1786334990297270 us; error 29 us; skew 500 ppm
Aug 10 04:09:50 dist-test-slave-8w9v krb5kdc[16241](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334990, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:09:50.299973 17233 init.cc:377] Logged in from keytab as kudu/127.31.32.3@KRBTEST.COM (short username kudu)
I20260810 04:09:50.300413 17233 webserver.cc:460] Webserver started at http://127.31.32.3:43127/ using document root <none> and password file <none>
I20260810 04:09:50.300593 17233 fs_manager.cc:359] Metadata directory not provided
I20260810 04:09:50.300642 17233 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:09:50.301831 17233 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:09:50.302452 17251 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:09:50.302628 17233 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:09:50.302718 17233 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/ts-2/data,/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/ts-2/wal
uuid: "9eda5e08e6434319b8b3626b9fabf245"
format_stamp: "Formatted at 2026-08-10 04:09:29 on dist-test-slave-8w9v"
server_key: "7b9b99a18e68ebcd40b887bca207e8fd"
server_key_iv: "d591e51a46ccd8da63898fbdcd65317b"
server_key_version: "encryptionkey@0"
I20260810 04:09:50.303046 17233 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:09:50.325439 17233 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:09:50.327353 17233 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:09:50.327522 17233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:09:50.327772 17233 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:09:50.328143 17233 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:09:50.328189 17233 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:09:50.328212 17233 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:09:50.328225 17233 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:09:50.334681 17233 rpc_server.cc:307] RPC server started. Bound to: 127.31.32.3:44889
I20260810 04:09:50.334749 17376 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.32.3:44889 every 8 connection(s)
I20260810 04:09:50.335110 17233 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/ts-2/data/info.pb
I20260810 04:09:50.339305 31872 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskws89Io/build/release/bin/kudu as pid 17233
Aug 10 04:09:50 dist-test-slave-8w9v krb5kdc[16241](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334990, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.3@KRBTEST.COM for kudu/127.31.32.62@KRBTEST.COM
I20260810 04:09:50.345340 17080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:50.335824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.3:46467 (local address 127.31.32.62:39943)
0810 04:09:50.335873 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0810 04:09:50.335875 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:09:50.336162 (+ 287us) server_negotiation.cc:407] Connection header received
0810 04:09:50.336633 (+ 471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:09:50.336635 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:09:50.336649 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:09:50.336740 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:09:50.337683 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:09:50.338174 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:09:50.339193 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:09:50.339388 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:09:50.339486 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:09:50.342096 (+ 2610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:09:50.342098 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:09:50.342098 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:09:50.342105 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:09:50.343661 (+ 1556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:09:50.344190 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:09:50.344191 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:09:50.344191 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:09:50.344243 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:09:50.344558 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:09:50.344559 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:09:50.344559 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:09:50.344710 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:09:50.344785 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0810 04:09:50.345154 (+ 369us) server_negotiation.cc:299] Negotiation successful
0810 04:09:50.345231 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260810 04:09:50.345417 17380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:50.335787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:39943 (local address 127.31.32.3:46467)
0810 04:09:50.335968 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0810 04:09:50.336024 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0810 04:09:50.336528 (+ 504us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:09:50.336820 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:09:50.336823 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:09:50.337394 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:09:50.337579 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:09:50.337584 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:09:50.338264 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:09:50.338265 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:09:50.339101 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:09:50.339104 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:09:50.339511 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:09:50.339512 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:09:50.339645 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:09:50.340250 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:09:50.340261 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:09:50.342019 (+ 1758us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:09:50.343752 (+ 1733us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:09:50.343759 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:09:50.343761 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:09:50.344077 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:09:50.344313 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:09:50.344314 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:09:50.344314 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:09:50.344458 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:09:50.344781 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:09:50.344786 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:09:50.344964 (+ 178us) client_negotiation.cc:769] Sending connection context
0810 04:09:50.345073 (+ 109us) client_negotiation.cc:240] Negotiation successful
0810 04:09:50.345223 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":105,"threads_started":1}
I20260810 04:09:50.346097 17377 heartbeater.cc:347] Connected to a master server at 127.31.32.62:39943
I20260810 04:09:50.346186 17377 heartbeater.cc:464] Registering TS with master...
I20260810 04:09:50.346381 17377 heartbeater.cc:511] Master 127.31.32.62:39943 requested a full tablet report, sending...
I20260810 04:09:50.346781 16872 ts_manager.cc:194] Registered new tserver with Master: 9eda5e08e6434319b8b3626b9fabf245 (127.31.32.3:44889)
I20260810 04:09:50.347142 16872 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.32.3@KRBTEST.COM'} at 127.31.32.3:46467
I20260810 04:09:50.347465 17382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:50.339767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41028 (local address 127.31.32.62:39943)
0810 04:09:50.339900 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0810 04:09:50.339902 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:09:50.339956 (+ 54us) server_negotiation.cc:407] Connection header received
0810 04:09:50.340008 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:09:50.340008 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:09:50.340020 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:09:50.340088 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:09:50.340921 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:09:50.341659 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:09:50.342795 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:09:50.342998 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:09:50.343075 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:09:50.344568 (+ 1493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:09:50.344568 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:09:50.344569 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:09:50.344575 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:09:50.346366 (+ 1791us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:09:50.346690 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:09:50.346691 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:09:50.346691 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:09:50.346755 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:09:50.346950 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:09:50.346951 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:09:50.346951 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:09:50.347060 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:09:50.347240 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0810 04:09:50.347277 (+ 37us) server_negotiation.cc:299] Negotiation successful
0810 04:09:50.347356 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":60,"threads_started":1}
I20260810 04:09:50.348093 31872 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 04:09:50.350699 17382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:50.348465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41044 (local address 127.31.32.62:39943)
0810 04:09:50.348511 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0810 04:09:50.348513 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:09:50.348623 (+ 110us) server_negotiation.cc:407] Connection header received
0810 04:09:50.348676 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:09:50.348676 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:09:50.348700 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:09:50.348754 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 04:09:50.348865 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:09:50.349275 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:09:50.349956 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:09:50.350130 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:09:50.350177 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:09:50.350473 (+ 296us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 04:09:50.350507 (+ 34us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0810 04:09:50.350625 (+ 118us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:41044: token expired
Metrics: {"server-negotiator.queue_time_us":29}
W20260810 04:09:50.350770 17382 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:41044: token expired
W20260810 04:09:50.350961 16795 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.31.32.62:39943)
I20260810 04:09:50.356353 17382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:50.351146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41058 (local address 127.31.32.62:39943)
0810 04:09:50.351196 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0810 04:09:50.351197 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 04:09:50.351205 (+ 8us) server_negotiation.cc:407] Connection header received
0810 04:09:50.351253 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:09:50.351253 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:09:50.351260 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:09:50.351303 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:09:50.351691 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:09:50.352107 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:09:50.352844 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:09:50.353008 (+ 164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:09:50.353055 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:09:50.354048 (+ 993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:09:50.354049 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:09:50.354049 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:09:50.354055 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:09:50.355470 (+ 1415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:09:50.355748 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:09:50.355748 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:09:50.355749 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:09:50.355797 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:09:50.355954 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:09:50.355954 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:09:50.355954 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:09:50.356044 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:09:50.356106 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0810 04:09:50.356244 (+ 138us) server_negotiation.cc:299] Negotiation successful
0810 04:09:50.356301 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260810 04:09:50.357451 16872 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41058:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 04:09:50.358940 16872 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 04:09:50.361129 17158 tablet_service.cc:1467] Processing CreateTablet for tablet b6d443aae643468f8a06461ea85fe196 (DEFAULT_TABLE table=test-table [id=e2f493d6597141f8a0cb6005ff3db942]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:09:50.361129 17004 tablet_service.cc:1467] Processing CreateTablet for tablet b6d443aae643468f8a06461ea85fe196 (DEFAULT_TABLE table=test-table [id=e2f493d6597141f8a0cb6005ff3db942]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:09:50.361323 17004 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6d443aae643468f8a06461ea85fe196. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:09:50.361323 17158 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6d443aae643468f8a06461ea85fe196. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:09:50.362968 17110 tablet_bootstrap.cc:492] T b6d443aae643468f8a06461ea85fe196 P 1fa692fb508741e1be5992f3fd536b48: Bootstrap starting.
I20260810 04:09:50.362986 16957 tablet_bootstrap.cc:492] T b6d443aae643468f8a06461ea85fe196 P 7b0dea1b85284b918eb1601d887ae01b: Bootstrap starting.
I20260810 04:09:50.363328 16957 tablet_bootstrap.cc:654] T b6d443aae643468f8a06461ea85fe196 P 7b0dea1b85284b918eb1601d887ae01b: Neither blocks nor log segments found. Creating new log.
I20260810 04:09:50.363402 17110 tablet_bootstrap.cc:654] T b6d443aae643468f8a06461ea85fe196 P 1fa692fb508741e1be5992f3fd536b48: Neither blocks nor log segments found. Creating new log.
I20260810 04:09:50.364111 17110 tablet_bootstrap.cc:492] T b6d443aae643468f8a06461ea85fe196 P 1fa692fb508741e1be5992f3fd536b48: No bootstrap required, opened a new log
I20260810 04:09:50.364189 17110 ts_tablet_manager.cc:1397] T b6d443aae643468f8a06461ea85fe196 P 1fa692fb508741e1be5992f3fd536b48: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260810 04:09:50.364298 16957 tablet_bootstrap.cc:492] T b6d443aae643468f8a06461ea85fe196 P 7b0dea1b85284b918eb1601d887ae01b: No bootstrap required, opened a new log
I20260810 04:09:50.364358 16957 ts_tablet_manager.cc:1397] T b6d443aae643468f8a06461ea85fe196 P 7b0dea1b85284b918eb1601d887ae01b: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260810 04:09:50.364495 17110 raft_consensus.cc:348] T b6d443aae643468f8a06461ea85fe196 P 1fa692fb508741e1be5992f3fd536b48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9eda5e08e6434319b8b3626b9fabf245" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 44889 } } peers { permanent_uuid: "7b0dea1b85284b918eb1601d887ae01b" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 46861 } } peers { permanent_uuid: "1fa692fb508741e1be5992f3fd536b48" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 45549 } }
I20260810 04:09:50.364526 16957 raft_consensus.cc:348] T b6d443aae643468f8a06461ea85fe196 P 7b0dea1b85284b918eb1601d887ae01b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9eda5e08e6434319b8b3626b9fabf245" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 44889 } } peers { permanent_uuid: "7b0dea1b85284b918eb1601d887ae01b" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 46861 } } peers { permanent_uuid: "1fa692fb508741e1be5992f3fd536b48" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 45549 } }
I20260810 04:09:50.364580 17110 raft_consensus.cc:374] T b6d443aae643468f8a06461ea85fe196 P 1fa692fb508741e1be5992f3fd536b48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:09:50.364622 17110 raft_consensus.cc:729] T b6d443aae643468f8a06461ea85fe196 P 1fa692fb508741e1be5992f3fd536b48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fa692fb508741e1be5992f3fd536b48, State: Initialized, Role: FOLLOWER
I20260810 04:09:50.364602 16957 raft_consensus.cc:374] T b6d443aae643468f8a06461ea85fe196 P 7b0dea1b85284b918eb1601d887ae01b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:09:50.364652 16957 raft_consensus.cc:729] T b6d443aae643468f8a06461ea85fe196 P 7b0dea1b85284b918eb1601d887ae01b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b0dea1b85284b918eb1601d887ae01b, State: Initialized, Role: FOLLOWER
I20260810 04:09:50.364698 17110 consensus_queue.cc:260] T b6d443aae643468f8a06461ea85fe196 P 1fa692fb508741e1be5992f3fd536b48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9eda5e08e6434319b8b3626b9fabf245" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 44889 } } peers { permanent_uuid: "7b0dea1b85284b918eb1601d887ae01b" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 46861 } } peers { permanent_uuid: "1fa692fb508741e1be5992f3fd536b48" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 45549 } }
I20260810 04:09:50.364712 16957 consensus_queue.cc:260] T b6d443aae643468f8a06461ea85fe196 P 7b0dea1b85284b918eb1601d887ae01b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9eda5e08e6434319b8b3626b9fabf245" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 44889 } } peers { permanent_uuid: "7b0dea1b85284b918eb1601d887ae01b" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 46861 } } peers { permanent_uuid: "1fa692fb508741e1be5992f3fd536b48" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 45549 } }
I20260810 04:09:50.364813 17110 ts_tablet_manager.cc:1428] T b6d443aae643468f8a06461ea85fe196 P 1fa692fb508741e1be5992f3fd536b48: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 04:09:50.364814 17084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:50.360806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.3:44889 (local address 127.0.0.1:56056)
0810 04:09:50.360864 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0810 04:09:50.360875 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 04:09:50.360909 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:09:50.361379 (+ 470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:09:50.361380 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:09:50.361382 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:09:50.361425 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:09:50.361426 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:09:50.362438 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:09:50.362439 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:09:50.363330 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:09:50.363333 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:09:50.364519 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:09:50.364520 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:09:50.364585 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:09:50.364611 (+ 26us) client_negotiation.cc:769] Sending connection context
0810 04:09:50.364691 (+ 80us) client_negotiation.cc:240] Negotiation successful
0810 04:09:50.364743 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260810 04:09:50.364832 16957 ts_tablet_manager.cc:1428] T b6d443aae643468f8a06461ea85fe196 P 7b0dea1b85284b918eb1601d887ae01b: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260810 04:09:50.364899 17384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:50.360924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56056 (local address 127.31.32.3:44889)
0810 04:09:50.361136 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0810 04:09:50.361141 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 04:09:50.361152 (+ 11us) server_negotiation.cc:407] Connection header received
0810 04:09:50.361197 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:09:50.361198 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:09:50.361232 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:09:50.361521 (+ 289us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:09:50.361630 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:09:50.362326 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:09:50.363507 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:09:50.364442 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:09:50.364517 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:09:50.364575 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0810 04:09:50.364715 (+ 140us) server_negotiation.cc:299] Negotiation successful
0810 04:09:50.364804 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":143,"threads_started":1}
I20260810 04:09:50.364943 17229 heartbeater.cc:503] Master 127.31.32.62:39943 was elected leader, sending a full tablet report...
I20260810 04:09:50.365630 17302 tablet_service.cc:1467] Processing CreateTablet for tablet b6d443aae643468f8a06461ea85fe196 (DEFAULT_TABLE table=test-table [id=e2f493d6597141f8a0cb6005ff3db942]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:09:50.366032 17302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6d443aae643468f8a06461ea85fe196. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:09:50.368963 17387 tablet_bootstrap.cc:492] T b6d443aae643468f8a06461ea85fe196 P 9eda5e08e6434319b8b3626b9fabf245: Bootstrap starting.
I20260810 04:09:50.369539 17387 tablet_bootstrap.cc:654] T b6d443aae643468f8a06461ea85fe196 P 9eda5e08e6434319b8b3626b9fabf245: Neither blocks nor log segments found. Creating new log.
I20260810 04:09:50.370020 17387 log.cc:824] T b6d443aae643468f8a06461ea85fe196 P 9eda5e08e6434319b8b3626b9fabf245: Log is configured to *not* fsync() on all Append() calls
I20260810 04:09:50.370716 17387 tablet_bootstrap.cc:492] T b6d443aae643468f8a06461ea85fe196 P 9eda5e08e6434319b8b3626b9fabf245: No bootstrap required, opened a new log
I20260810 04:09:50.370801 17387 ts_tablet_manager.cc:1397] T b6d443aae643468f8a06461ea85fe196 P 9eda5e08e6434319b8b3626b9fabf245: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 04:09:50.372418 17387 raft_consensus.cc:348] T b6d443aae643468f8a06461ea85fe196 P 9eda5e08e6434319b8b3626b9fabf245 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9eda5e08e6434319b8b3626b9fabf245" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 44889 } } peers { permanent_uuid: "7b0dea1b85284b918eb1601d887ae01b" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 46861 } } peers { permanent_uuid: "1fa692fb508741e1be5992f3fd536b48" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 45549 } }
I20260810 04:09:50.372570 17387 raft_consensus.cc:374] T b6d443aae643468f8a06461ea85fe196 P 9eda5e08e6434319b8b3626b9fabf245 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:09:50.372618 17387 raft_consensus.cc:729] T b6d443aae643468f8a06461ea85fe196 P 9eda5e08e6434319b8b3626b9fabf245 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9eda5e08e6434319b8b3626b9fabf245, State: Initialized, Role: FOLLOWER
I20260810 04:09:50.372735 17387 consensus_queue.cc:260] T b6d443aae643468f8a06461ea85fe196 P 9eda5e08e6434319b8b3626b9fabf245 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9eda5e08e6434319b8b3626b9fabf245" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 44889 } } peers { permanent_uuid: "7b0dea1b85284b918eb1601d887ae01b" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 46861 } } peers { permanent_uuid: "1fa692fb508741e1be5992f3fd536b48" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 45549 } }
I20260810 04:09:50.372982 17387 ts_tablet_manager.cc:1428] T b6d443aae643468f8a06461ea85fe196 P 9eda5e08e6434319b8b3626b9fabf245: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260810 04:09:50.373041 17377 heartbeater.cc:503] Master 127.31.32.62:39943 was elected leader, sending a full tablet report...
I20260810 04:09:50.376475 17083 raft_consensus.cc:482] T b6d443aae643468f8a06461ea85fe196 P 7b0dea1b85284b918eb1601d887ae01b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 04:09:50.376591 17083 raft_consensus.cc:504] T b6d443aae643468f8a06461ea85fe196 P 7b0dea1b85284b918eb1601d887ae01b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9eda5e08e6434319b8b3626b9fabf245" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 44889 } } peers { permanent_uuid: "7b0dea1b85284b918eb1601d887ae01b" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 46861 } } peers { permanent_uuid: "1fa692fb508741e1be5992f3fd536b48" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 45549 } }
I20260810 04:09:50.376945 17083 leader_election.cc:288] T b6d443aae643468f8a06461ea85fe196 P 7b0dea1b85284b918eb1601d887ae01b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9eda5e08e6434319b8b3626b9fabf245 (127.31.32.3:44889), 1fa692fb508741e1be5992f3fd536b48 (127.31.32.2:45549)
I20260810 04:09:50.380650 17384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:50.377192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.1:34717 (local address 127.31.32.3:44889)
0810 04:09:50.377343 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0810 04:09:50.377345 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:09:50.377353 (+ 8us) server_negotiation.cc:407] Connection header received
0810 04:09:50.377380 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:09:50.377381 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:09:50.377389 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:09:50.377434 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:09:50.377592 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:09:50.378322 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:09:50.379689 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:09:50.380265 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:09:50.380341 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:09:50.380352 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0810 04:09:50.380555 (+ 203us) server_negotiation.cc:299] Negotiation successful
0810 04:09:50.380591 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129}
I20260810 04:09:50.380656 17234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:50.377298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.1:48237 (local address 127.31.32.2:45549)
0810 04:09:50.377350 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0810 04:09:50.377352 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:09:50.377384 (+ 32us) server_negotiation.cc:407] Connection header received
0810 04:09:50.377484 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:09:50.377485 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:09:50.377506 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:09:50.377584 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:09:50.377732 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:09:50.378437 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:09:50.379690 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:09:50.380337 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:09:50.380401 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:09:50.380448 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0810 04:09:50.380555 (+ 107us) server_negotiation.cc:299] Negotiation successful
0810 04:09:50.380594 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260810 04:09:50.380689 17079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:50.377095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.3:44889 (local address 127.31.32.1:34717)
0810 04:09:50.377151 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0810 04:09:50.377163 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 04:09:50.377225 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:09:50.377454 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:09:50.377456 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:09:50.377457 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:09:50.377523 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:09:50.377525 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:09:50.378397 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:09:50.378398 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:09:50.379597 (+ 1199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:09:50.379599 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:09:50.380354 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:09:50.380355 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:09:50.380448 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:09:50.380473 (+ 25us) client_negotiation.cc:769] Sending connection context
0810 04:09:50.380536 (+ 63us) client_negotiation.cc:240] Negotiation successful
0810 04:09:50.380592 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33,"mutex_wait_us":14}
I20260810 04:09:50.380708 17390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:50.377194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.2:45549 (local address 127.31.32.1:48237)
0810 04:09:50.377334 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0810 04:09:50.377344 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 04:09:50.377397 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:09:50.377600 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:09:50.377601 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:09:50.377602 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:09:50.377672 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:09:50.377673 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:09:50.378505 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:09:50.378505 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:09:50.379597 (+ 1092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:09:50.379599 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:09:50.380422 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:09:50.380423 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:09:50.380475 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:09:50.380478 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 04:09:50.380536 (+ 58us) client_negotiation.cc:240] Negotiation successful
0810 04:09:50.380592 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"mutex_wait_us":12,"thread_start_us":91,"threads_started":1}
I20260810 04:09:50.381014 17324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6d443aae643468f8a06461ea85fe196" candidate_uuid: "7b0dea1b85284b918eb1601d887ae01b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9eda5e08e6434319b8b3626b9fabf245" is_pre_election: true
I20260810 04:09:50.381038 17180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6d443aae643468f8a06461ea85fe196" candidate_uuid: "7b0dea1b85284b918eb1601d887ae01b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fa692fb508741e1be5992f3fd536b48" is_pre_election: true
I20260810 04:09:50.381145 17324 raft_consensus.cc:2454] T b6d443aae643468f8a06461ea85fe196 P 9eda5e08e6434319b8b3626b9fabf245 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b0dea1b85284b918eb1601d887ae01b in term 0.
I20260810 04:09:50.381152 17180 raft_consensus.cc:2454] T b6d443aae643468f8a06461ea85fe196 P 1fa692fb508741e1be5992f3fd536b48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b0dea1b85284b918eb1601d887ae01b in term 0.
I20260810 04:09:50.381356 16951 leader_election.cc:302] T b6d443aae643468f8a06461ea85fe196 P 7b0dea1b85284b918eb1601d887ae01b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fa692fb508741e1be5992f3fd536b48, 7b0dea1b85284b918eb1601d887ae01b; no voters:
I20260810 04:09:50.381572 17083 raft_consensus.cc:2781] T b6d443aae643468f8a06461ea85fe196 P 7b0dea1b85284b918eb1601d887ae01b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 04:09:50.381646 17083 raft_consensus.cc:482] T b6d443aae643468f8a06461ea85fe196 P 7b0dea1b85284b918eb1601d887ae01b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 04:09:50.381664 17083 raft_consensus.cc:3037] T b6d443aae643468f8a06461ea85fe196 P 7b0dea1b85284b918eb1601d887ae01b [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:09:50.382330 17083 raft_consensus.cc:504] T b6d443aae643468f8a06461ea85fe196 P 7b0dea1b85284b918eb1601d887ae01b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9eda5e08e6434319b8b3626b9fabf245" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 44889 } } peers { permanent_uuid: "7b0dea1b85284b918eb1601d887ae01b" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 46861 } } peers { permanent_uuid: "1fa692fb508741e1be5992f3fd536b48" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 45549 } }
I20260810 04:09:50.382462 17083 leader_election.cc:288] T b6d443aae643468f8a06461ea85fe196 P 7b0dea1b85284b918eb1601d887ae01b [CANDIDATE]: Term 1 election: Requested vote from peers 9eda5e08e6434319b8b3626b9fabf245 (127.31.32.3:44889), 1fa692fb508741e1be5992f3fd536b48 (127.31.32.2:45549)
I20260810 04:09:50.382678 17324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6d443aae643468f8a06461ea85fe196" candidate_uuid: "7b0dea1b85284b918eb1601d887ae01b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9eda5e08e6434319b8b3626b9fabf245"
I20260810 04:09:50.382725 17180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6d443aae643468f8a06461ea85fe196" candidate_uuid: "7b0dea1b85284b918eb1601d887ae01b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fa692fb508741e1be5992f3fd536b48"
I20260810 04:09:50.382755 17324 raft_consensus.cc:3037] T b6d443aae643468f8a06461ea85fe196 P 9eda5e08e6434319b8b3626b9fabf245 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:09:50.382831 17180 raft_consensus.cc:3037] T b6d443aae643468f8a06461ea85fe196 P 1fa692fb508741e1be5992f3fd536b48 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:09:50.383487 17324 raft_consensus.cc:2454] T b6d443aae643468f8a06461ea85fe196 P 9eda5e08e6434319b8b3626b9fabf245 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b0dea1b85284b918eb1601d887ae01b in term 1.
I20260810 04:09:50.383567 17180 raft_consensus.cc:2454] T b6d443aae643468f8a06461ea85fe196 P 1fa692fb508741e1be5992f3fd536b48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b0dea1b85284b918eb1601d887ae01b in term 1.
I20260810 04:09:50.383666 16951 leader_election.cc:302] T b6d443aae643468f8a06461ea85fe196 P 7b0dea1b85284b918eb1601d887ae01b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b0dea1b85284b918eb1601d887ae01b, 9eda5e08e6434319b8b3626b9fabf245; no voters:
I20260810 04:09:50.383770 17083 raft_consensus.cc:2781] T b6d443aae643468f8a06461ea85fe196 P 7b0dea1b85284b918eb1601d887ae01b [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:09:50.383904 17083 raft_consensus.cc:686] T b6d443aae643468f8a06461ea85fe196 P 7b0dea1b85284b918eb1601d887ae01b [term 1 LEADER]: Becoming Leader. State: Replica: 7b0dea1b85284b918eb1601d887ae01b, State: Running, Role: LEADER
I20260810 04:09:50.383957 17083 consensus_queue.cc:237] T b6d443aae643468f8a06461ea85fe196 P 7b0dea1b85284b918eb1601d887ae01b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9eda5e08e6434319b8b3626b9fabf245" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 44889 } } peers { permanent_uuid: "7b0dea1b85284b918eb1601d887ae01b" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 46861 } } peers { permanent_uuid: "1fa692fb508741e1be5992f3fd536b48" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 45549 } }
I20260810 04:09:50.384706 16871 catalog_manager.cc:5495] T b6d443aae643468f8a06461ea85fe196 P 7b0dea1b85284b918eb1601d887ae01b reported cstate change: term changed from 0 to 1, leader changed from <none> to 7b0dea1b85284b918eb1601d887ae01b (127.31.32.1). New cstate: current_term: 1 leader_uuid: "7b0dea1b85284b918eb1601d887ae01b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9eda5e08e6434319b8b3626b9fabf245" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 44889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b0dea1b85284b918eb1601d887ae01b" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 46861 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1fa692fb508741e1be5992f3fd536b48" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 45549 } health_report { overall_health: UNKNOWN } } }
I20260810 04:09:50.394490 17085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:50.392436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56310 (local address 127.31.32.1:46861)
0810 04:09:50.392495 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0810 04:09:50.392497 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:09:50.392504 (+ 7us) server_negotiation.cc:407] Connection header received
0810 04:09:50.392557 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:09:50.392558 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:09:50.392572 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:09:50.392627 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 04:09:50.392730 (+ 103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:09:50.393182 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:09:50.393884 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:09:50.394073 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:09:50.394125 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:09:50.394240 (+ 115us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 04:09:50.394324 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 04:09:50.394383 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0810 04:09:50.394425 (+ 42us) server_negotiation.cc:299] Negotiation successful
0810 04:09:50.394444 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260810 04:09:50.395783 17180 raft_consensus.cc:1261] T b6d443aae643468f8a06461ea85fe196 P 1fa692fb508741e1be5992f3fd536b48 [term 1 FOLLOWER]: Refusing update from remote peer 7b0dea1b85284b918eb1601d887ae01b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:09:50.395783 17324 raft_consensus.cc:1261] T b6d443aae643468f8a06461ea85fe196 P 9eda5e08e6434319b8b3626b9fabf245 [term 1 FOLLOWER]: Refusing update from remote peer 7b0dea1b85284b918eb1601d887ae01b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:09:50.396124 17391 consensus_queue.cc:1035] T b6d443aae643468f8a06461ea85fe196 P 7b0dea1b85284b918eb1601d887ae01b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fa692fb508741e1be5992f3fd536b48" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 45549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:09:50.396399 17083 consensus_queue.cc:1035] T b6d443aae643468f8a06461ea85fe196 P 7b0dea1b85284b918eb1601d887ae01b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9eda5e08e6434319b8b3626b9fabf245" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 44889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:09:50.398357 17395 mvcc.cc:205] Tried to move back new op lower bound from 7316828120658935808 to 7316828120613367808. Current Snapshot: MvccSnapshot[applied={T|T < 7316828120658935808}]
I20260810 04:09:50.400086 16871 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41058:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 04:09:50.400216 16871 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41058:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 04:09:50.400496 17396 mvcc.cc:205] Tried to move back new op lower bound from 7316828120658935808 to 7316828120613367808. Current Snapshot: MvccSnapshot[applied={T|T < 7316828120658935808}]
I20260810 04:09:50.400894 16871 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 39ad49896b2d4abf8bf3e221466c2cf5: Sending DeleteTablet for 3 replicas of tablet b6d443aae643468f8a06461ea85fe196
I20260810 04:09:50.401139 17158 tablet_service.cc:1514] Processing DeleteTablet for tablet b6d443aae643468f8a06461ea85fe196 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:09:50 UTC) from {username='kudu', principal='kudu/127.31.32.62@KRBTEST.COM'} at 127.0.0.1:59496
I20260810 04:09:50.401155 17004 tablet_service.cc:1514] Processing DeleteTablet for tablet b6d443aae643468f8a06461ea85fe196 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:09:50 UTC) from {username='kudu', principal='kudu/127.31.32.62@KRBTEST.COM'} at 127.0.0.1:56306
I20260810 04:09:50.401237 17235 tablet_replica.cc:330] T b6d443aae643468f8a06461ea85fe196 P 1fa692fb508741e1be5992f3fd536b48: stopping tablet replica
I20260810 04:09:50.401247 17086 tablet_replica.cc:330] T b6d443aae643468f8a06461ea85fe196 P 7b0dea1b85284b918eb1601d887ae01b: stopping tablet replica
I20260810 04:09:50.401292 17235 raft_consensus.cc:2229] T b6d443aae643468f8a06461ea85fe196 P 1fa692fb508741e1be5992f3fd536b48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 04:09:50.401299 17086 raft_consensus.cc:2229] T b6d443aae643468f8a06461ea85fe196 P 7b0dea1b85284b918eb1601d887ae01b [term 1 LEADER]: Raft consensus shutting down.
I20260810 04:09:50.401357 17235 raft_consensus.cc:2258] T b6d443aae643468f8a06461ea85fe196 P 1fa692fb508741e1be5992f3fd536b48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:09:50.401384 17086 raft_consensus.cc:2258] T b6d443aae643468f8a06461ea85fe196 P 7b0dea1b85284b918eb1601d887ae01b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:09:50.401540 17302 tablet_service.cc:1514] Processing DeleteTablet for tablet b6d443aae643468f8a06461ea85fe196 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:09:50 UTC) from {username='kudu', principal='kudu/127.31.32.62@KRBTEST.COM'} at 127.0.0.1:56056
I20260810 04:09:50.401731 17301 tablet_service.cc:1514] Processing DeleteTablet for tablet b6d443aae643468f8a06461ea85fe196 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:09:50 UTC) from {username='kudu', principal='kudu/127.31.32.62@KRBTEST.COM'} at 127.0.0.1:56056
I20260810 04:09:50.401743 17235 ts_tablet_manager.cc:1905] T b6d443aae643468f8a06461ea85fe196 P 1fa692fb508741e1be5992f3fd536b48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:09:50.401746 17086 ts_tablet_manager.cc:1905] T b6d443aae643468f8a06461ea85fe196 P 7b0dea1b85284b918eb1601d887ae01b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:09:50.402020 31872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskws89Io/build/release/bin/kudu with pid 16913
I20260810 04:09:50.403640 17235 ts_tablet_manager.cc:1918] T b6d443aae643468f8a06461ea85fe196 P 1fa692fb508741e1be5992f3fd536b48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 04:09:50.403714 17235 log.cc:1196] T b6d443aae643468f8a06461ea85fe196 P 1fa692fb508741e1be5992f3fd536b48: Deleting WAL directory at /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/ts-1/wal/wals/b6d443aae643468f8a06461ea85fe196
I20260810 04:09:50.404011 17235 ts_tablet_manager.cc:1939] T b6d443aae643468f8a06461ea85fe196 P 1fa692fb508741e1be5992f3fd536b48: Deleting consensus metadata
I20260810 04:09:50.404439 16853 catalog_manager.cc:4841] TS 1fa692fb508741e1be5992f3fd536b48 (127.31.32.2:45549): tablet b6d443aae643468f8a06461ea85fe196 (table test-table [id=e2f493d6597141f8a0cb6005ff3db942]) successfully deleted
I20260810 04:09:50.411950 17403 tablet_replica.cc:330] T b6d443aae643468f8a06461ea85fe196 P 9eda5e08e6434319b8b3626b9fabf245: stopping tablet replica
I20260810 04:09:50.412058 17403 raft_consensus.cc:2229] T b6d443aae643468f8a06461ea85fe196 P 9eda5e08e6434319b8b3626b9fabf245 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 04:09:50.412286 17403 raft_consensus.cc:2258] T b6d443aae643468f8a06461ea85fe196 P 9eda5e08e6434319b8b3626b9fabf245 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:09:50.413209 17403 ts_tablet_manager.cc:1905] T b6d443aae643468f8a06461ea85fe196 P 9eda5e08e6434319b8b3626b9fabf245: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:09:50.414609 17403 ts_tablet_manager.cc:1918] T b6d443aae643468f8a06461ea85fe196 P 9eda5e08e6434319b8b3626b9fabf245: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 04:09:50.414697 17403 log.cc:1196] T b6d443aae643468f8a06461ea85fe196 P 9eda5e08e6434319b8b3626b9fabf245: Deleting WAL directory at /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/ts-2/wal/wals/b6d443aae643468f8a06461ea85fe196
I20260810 04:09:50.414975 17403 ts_tablet_manager.cc:1939] T b6d443aae643468f8a06461ea85fe196 P 9eda5e08e6434319b8b3626b9fabf245: Deleting consensus metadata
I20260810 04:09:50.415505 16853 catalog_manager.cc:4841] TS 9eda5e08e6434319b8b3626b9fabf245 (127.31.32.3:44889): tablet b6d443aae643468f8a06461ea85fe196 (table test-table [id=e2f493d6597141f8a0cb6005ff3db942]) successfully deleted
W20260810 04:09:50.415652 16853 catalog_manager.cc:4808] TS 9eda5e08e6434319b8b3626b9fabf245 (127.31.32.3:44889): delete failed for tablet b6d443aae643468f8a06461ea85fe196 because the tablet was not found. No further retry: Not found: Tablet not found: b6d443aae643468f8a06461ea85fe196
W20260810 04:09:50.415940 16854 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.32.1:46861 (error 108)
I20260810 04:09:50.416309 31872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskws89Io/build/release/bin/kudu with pid 17082
I20260810 04:09:50.416501 17084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:09:50.416368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.1:46861 (local address 127.0.0.1:56318)
0810 04:09:50.416430 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0810 04:09:50.416466 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.1:46861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260810 04:09:50.419404 16854 catalog_manager.cc:4584] TS 7b0dea1b85284b918eb1601d887ae01b (127.31.32.1:46861): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b6d443aae643468f8a06461ea85fe196: Network error: Client connection negotiation failed: client connection to 127.31.32.1:46861: connect: Connection refused (error 111)
I20260810 04:09:50.423244 31872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskws89Io/build/release/bin/kudu with pid 17233
I20260810 04:09:50.430909 31872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskws89Io/build/release/bin/kudu with pid 16834
I20260810 04:10:10.437690 31872 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskws89Io/build/release/bin/kudu
/tmp/dist-test-taskws89Io/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.32.62:39943
--webserver_interface=127.31.32.62
--webserver_port=42757
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.62.keytab
--principal=kudu/127.31.32.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.32.20:40835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.32.62:39943
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:10:10.520815 17406 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:10:10.521063 17406 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:10:10.521099 17406 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:10:10.522224 17406 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260810 04:10:10.522294 17406 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260810 04:10:10.522315 17406 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260810 04:10:10.522343 17406 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:10:10.522375 17406 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:10:10.522400 17406 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:10:10.522431 17406 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:10:10.522454 17406 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:10:10.523969 17406 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.32.20:40835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.31.32.62:39943
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.62.keytab
--principal=kudu/127.31.32.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.32.62:39943
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.32.62
--webserver_port=42757
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17406
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:10:10.524286 17406 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:10:10.524528 17406 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 04:10:10.529858 17406 server_base.cc:1034] running on GCE node
W20260810 04:10:10.529899 17415 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:10:10.529809 17417 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:10:10.529819 17414 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:10:10.530421 17406 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:10:10.530687 17406 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:10:10.531842 17406 hybrid_clock.cc:648] HybridClock initialized: now 1786335010531817 us; error 38 us; skew 500 ppm
Aug 10 04:10:10 dist-test-slave-8w9v krb5kdc[16241](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335010, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:10:10.538861 17406 init.cc:377] Logged in from keytab as kudu/127.31.32.62@KRBTEST.COM (short username kudu)
I20260810 04:10:10.539279 17406 webserver.cc:460] Webserver started at http://127.31.32.62:42757/ using document root <none> and password file <none>
I20260810 04:10:10.539522 17406 fs_manager.cc:359] Metadata directory not provided
I20260810 04:10:10.539575 17406 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:10:10.541071 17406 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.000s
I20260810 04:10:10.541868 17425 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:10:10.542088 17406 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:10:10.542182 17406 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/master-0/data,/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/master-0/wal
uuid: "39ad49896b2d4abf8bf3e221466c2cf5"
format_stamp: "Formatted at 2026-08-10 04:09:28 on dist-test-slave-8w9v"
server_key: "4465989c929bab8a58259d84eea563fe"
server_key_iv: "8cc3a42036dbb78e710b5de52047c8df"
server_key_version: "encryptionkey@0"
I20260810 04:10:10.542546 17406 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:10:10.550734 17406 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:10:10.552536 17406 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:10:10.552660 17406 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:10:10.556914 17406 rpc_server.cc:307] RPC server started. Bound to: 127.31.32.62:39943
I20260810 04:10:10.556991 17485 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.32.62:39943 every 8 connection(s)
I20260810 04:10:10.557363 17406 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/master-0/data/info.pb
I20260810 04:10:10.558859 17486 sys_catalog.cc:263] Verifying existing consensus state
I20260810 04:10:10.559533 17486 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39ad49896b2d4abf8bf3e221466c2cf5: Bootstrap starting.
I20260810 04:10:10.563722 31872 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskws89Io/build/release/bin/kudu as pid 17406
I20260810 04:10:10.564039 31872 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskws89Io/build/release/bin/kudu
/tmp/dist-test-taskws89Io/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.32.1:46861
--local_ip_for_outbound_sockets=127.31.32.1
--tserver_master_addrs=127.31.32.62:39943
--webserver_port=45745
--webserver_interface=127.31.32.1
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.1.keytab
--principal=kudu/127.31.32.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.32.20:40835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 04:10:10.568953 17486 log.cc:824] T 00000000000000000000000000000000 P 39ad49896b2d4abf8bf3e221466c2cf5: Log is configured to *not* fsync() on all Append() calls
I20260810 04:10:10.571175 17486 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39ad49896b2d4abf8bf3e221466c2cf5: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 04:10:10.571485 17486 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39ad49896b2d4abf8bf3e221466c2cf5: Bootstrap complete.
I20260810 04:10:10.573781 17486 raft_consensus.cc:348] T 00000000000000000000000000000000 P 39ad49896b2d4abf8bf3e221466c2cf5 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39ad49896b2d4abf8bf3e221466c2cf5" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 39943 } }
I20260810 04:10:10.574038 17486 raft_consensus.cc:729] T 00000000000000000000000000000000 P 39ad49896b2d4abf8bf3e221466c2cf5 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39ad49896b2d4abf8bf3e221466c2cf5, State: Initialized, Role: FOLLOWER
I20260810 04:10:10.574249 17486 consensus_queue.cc:260] T 00000000000000000000000000000000 P 39ad49896b2d4abf8bf3e221466c2cf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39ad49896b2d4abf8bf3e221466c2cf5" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 39943 } }
I20260810 04:10:10.574344 17486 raft_consensus.cc:388] T 00000000000000000000000000000000 P 39ad49896b2d4abf8bf3e221466c2cf5 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 04:10:10.574401 17486 raft_consensus.cc:482] T 00000000000000000000000000000000 P 39ad49896b2d4abf8bf3e221466c2cf5 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 04:10:10.574450 17486 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 39ad49896b2d4abf8bf3e221466c2cf5 [term 2 FOLLOWER]: Advancing to term 3
I20260810 04:10:10.575428 17486 raft_consensus.cc:504] T 00000000000000000000000000000000 P 39ad49896b2d4abf8bf3e221466c2cf5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39ad49896b2d4abf8bf3e221466c2cf5" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 39943 } }
I20260810 04:10:10.575548 17486 leader_election.cc:302] T 00000000000000000000000000000000 P 39ad49896b2d4abf8bf3e221466c2cf5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 39ad49896b2d4abf8bf3e221466c2cf5; no voters:
I20260810 04:10:10.575763 17486 leader_election.cc:288] T 00000000000000000000000000000000 P 39ad49896b2d4abf8bf3e221466c2cf5 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260810 04:10:10.575798 17490 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 39ad49896b2d4abf8bf3e221466c2cf5 [term 3 FOLLOWER]: Leader election won for term 3
I20260810 04:10:10.575925 17490 raft_consensus.cc:686] T 00000000000000000000000000000000 P 39ad49896b2d4abf8bf3e221466c2cf5 [term 3 LEADER]: Becoming Leader. State: Replica: 39ad49896b2d4abf8bf3e221466c2cf5, State: Running, Role: LEADER
I20260810 04:10:10.576033 17490 consensus_queue.cc:237] T 00000000000000000000000000000000 P 39ad49896b2d4abf8bf3e221466c2cf5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39ad49896b2d4abf8bf3e221466c2cf5" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 39943 } }
I20260810 04:10:10.576073 17486 sys_catalog.cc:564] T 00000000000000000000000000000000 P 39ad49896b2d4abf8bf3e221466c2cf5 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:10:10.576355 17491 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39ad49896b2d4abf8bf3e221466c2cf5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "39ad49896b2d4abf8bf3e221466c2cf5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39ad49896b2d4abf8bf3e221466c2cf5" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 39943 } } }
I20260810 04:10:10.576563 17491 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39ad49896b2d4abf8bf3e221466c2cf5 [sys.catalog]: This master's current role is: LEADER
I20260810 04:10:10.576363 17490 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39ad49896b2d4abf8bf3e221466c2cf5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 39ad49896b2d4abf8bf3e221466c2cf5. Latest consensus state: current_term: 3 leader_uuid: "39ad49896b2d4abf8bf3e221466c2cf5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39ad49896b2d4abf8bf3e221466c2cf5" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 39943 } } }
I20260810 04:10:10.576771 17490 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39ad49896b2d4abf8bf3e221466c2cf5 [sys.catalog]: This master's current role is: LEADER
I20260810 04:10:10.576819 17497 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 04:10:10.577658 17497 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 04:10:10.577867 17497 catalog_manager.cc:1259] Loaded cluster ID: 0bedc7d1051749e099934262464f9bde
I20260810 04:10:10.578078 17497 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 04:10:10.579304 17497 catalog_manager.cc:1504] Loading token signing keys...
I20260810 04:10:10.579533 17497 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 39ad49896b2d4abf8bf3e221466c2cf5: Loaded TSK: 0
I20260810 04:10:10.579631 17497 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 39ad49896b2d4abf8bf3e221466c2cf5: Loaded TSK: 1
F20260810 04:10:10.579936 17497 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f5a5b1259ad google::LogMessage::Fail() at ??:0
@ 0x7f5a5b12888a google::LogMessage::SendToLog() at ??:0
@ 0x7f5a5b1254de google::LogMessage::Flush() at ??:0
@ 0x7f5a5b126a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f5a5d892104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f5a5d88fd41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f5a630831cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f5a6308348e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f5a63083594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f5a63049c48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f5a63053ad0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f5a5cff30d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f5a5cfe8954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f5a5de4c6db start_thread at ??:0
@ 0x7f5a59c2361f clone at ??:0
W20260810 04:10:10.666668 17488 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:10:10.666878 17488 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:10:10.666903 17488 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:10:10.668049 17488 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:10:10.668089 17488 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:10:10.668115 17488 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.32.1
I20260810 04:10:10.669613 17488 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.32.20:40835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.1.keytab
--principal=kudu/127.31.32.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.32.1:46861
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.32.1
--webserver_port=45745
--tserver_master_addrs=127.31.32.62:39943
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17488
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.32.1
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:10:10.669896 17488 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:10:10.670128 17488 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:10:10.676258 17535 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:10:10.676246 17536 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:10:10.676473 17488 server_base.cc:1034] running on GCE node
W20260810 04:10:10.676620 17538 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:10:10.676808 17488 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:10:10.677063 17488 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:10:10.678225 17488 hybrid_clock.cc:648] HybridClock initialized: now 1786335010678212 us; error 31 us; skew 500 ppm
Aug 10 04:10:10 dist-test-slave-8w9v krb5kdc[16241](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335010, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:10:10.680917 17488 init.cc:377] Logged in from keytab as kudu/127.31.32.1@KRBTEST.COM (short username kudu)
I20260810 04:10:10.681422 17488 webserver.cc:460] Webserver started at http://127.31.32.1:45745/ using document root <none> and password file <none>
I20260810 04:10:10.681686 17488 fs_manager.cc:359] Metadata directory not provided
I20260810 04:10:10.681753 17488 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:10:10.683144 17488 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:10:10.683837 17545 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:10:10.683993 17488 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260810 04:10:10.684098 17488 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/ts-0/data,/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/ts-0/wal
uuid: "7b0dea1b85284b918eb1601d887ae01b"
format_stamp: "Formatted at 2026-08-10 04:09:28 on dist-test-slave-8w9v"
server_key: "b5b53b670d118fb11fea1357903cd359"
server_key_iv: "a2b1e2ab3dd73110d77044d83054bdc6"
server_key_version: "encryptionkey@0"
I20260810 04:10:10.684476 17488 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:10:10.702183 17488 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:10:10.704532 17488 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:10:10.704682 17488 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:10:10.704988 17488 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:10:10.705615 17552 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260810 04:10:10.707736 17552 ts_tablet_manager.cc:1852] T b6d443aae643468f8a06461ea85fe196 P 7b0dea1b85284b918eb1601d887ae01b: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260810 04:10:10.708160 17552 ts_tablet_manager.cc:1905] T b6d443aae643468f8a06461ea85fe196 P 7b0dea1b85284b918eb1601d887ae01b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:10:10.711012 17552 ts_tablet_manager.cc:1918] T b6d443aae643468f8a06461ea85fe196 P 7b0dea1b85284b918eb1601d887ae01b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 04:10:10.711150 17552 log.cc:1196] T b6d443aae643468f8a06461ea85fe196 P 7b0dea1b85284b918eb1601d887ae01b: Deleting WAL directory at /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/ts-0/wal/wals/b6d443aae643468f8a06461ea85fe196
I20260810 04:10:10.711506 17552 ts_tablet_manager.cc:1939] T b6d443aae643468f8a06461ea85fe196 P 7b0dea1b85284b918eb1601d887ae01b: Deleting consensus metadata
I20260810 04:10:10.711750 17488 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260810 04:10:10.711865 17488 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260810 04:10:10.711947 17488 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:10:10.712008 17488 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:10:10.719422 17488 rpc_server.cc:307] RPC server started. Bound to: 127.31.32.1:46861
I20260810 04:10:10.719504 17676 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.32.1:46861 every 8 connection(s)
I20260810 04:10:10.719811 17488 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/ts-0/data/info.pb
I20260810 04:10:10.720506 31872 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskws89Io/build/release/bin/kudu as pid 17488
I20260810 04:10:10.720810 31872 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskws89Io/build/release/bin/kudu
/tmp/dist-test-taskws89Io/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.32.2:45549
--local_ip_for_outbound_sockets=127.31.32.2
--tserver_master_addrs=127.31.32.62:39943
--webserver_port=39677
--webserver_interface=127.31.32.2
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.2.keytab
--principal=kudu/127.31.32.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.32.20:40835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:10:10.779233 17679 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:10:10.720587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:39943 (local address 127.31.32.1:37899)
0810 04:10:10.720842 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0810 04:10:10.720886 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 04:10:10.721550 (+ 664us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:10:10.779027 (+ 57477us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.62:39943: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":91,"threads_started":1}
W20260810 04:10:10.779510 17677 heartbeater.cc:644] Failed to heartbeat to 127.31.32.62:39943 (0 consecutive failures): Network error: Failed to ping master at 127.31.32.62:39943: Client connection negotiation failed: client connection to 127.31.32.62:39943: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260810 04:10:10.779898 17679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:10.779781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:39943 (local address 127.31.32.1:39211)
0810 04:10:10.779837 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0810 04:10:10.779881 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.62:39943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260810 04:10:10.780236 17679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:10.780157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:39943 (local address 127.31.32.1:43273)
0810 04:10:10.780191 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0810 04:10:10.780216 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.62:39943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
W20260810 04:10:10.780345 17677 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 04:10:10.808517 17680 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:10:10.808736 17680 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:10:10.808774 17680 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:10:10.809943 17680 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:10:10.810009 17680 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:10:10.810048 17680 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.32.2
I20260810 04:10:10.811480 17680 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.32.20:40835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.2.keytab
--principal=kudu/127.31.32.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.32.2:45549
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.32.2
--webserver_port=39677
--tserver_master_addrs=127.31.32.62:39943
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17680
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.32.2
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:10:10.811735 17680 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:10:10.811978 17680 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:10:10.817111 17687 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:10:10.817123 17689 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:10:10.817145 17686 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:10:10.817233 17680 server_base.cc:1034] running on GCE node
I20260810 04:10:10.817482 17680 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:10:10.817759 17680 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:10:10.818940 17680 hybrid_clock.cc:648] HybridClock initialized: now 1786335010818928 us; error 46 us; skew 500 ppm
Aug 10 04:10:10 dist-test-slave-8w9v krb5kdc[16241](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335010, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:10:10.821532 17680 init.cc:377] Logged in from keytab as kudu/127.31.32.2@KRBTEST.COM (short username kudu)
I20260810 04:10:10.821949 17680 webserver.cc:460] Webserver started at http://127.31.32.2:39677/ using document root <none> and password file <none>
I20260810 04:10:10.822193 17680 fs_manager.cc:359] Metadata directory not provided
I20260810 04:10:10.822263 17680 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:10:10.823623 17680 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:10:10.824261 17696 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:10:10.824415 17680 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 04:10:10.824503 17680 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/ts-1/data,/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/ts-1/wal
uuid: "1fa692fb508741e1be5992f3fd536b48"
format_stamp: "Formatted at 2026-08-10 04:09:29 on dist-test-slave-8w9v"
server_key: "64a1f9ecbd6b76abc5e22845cee6b701"
server_key_iv: "8518bbb579c2bd18ac4604451e2b4cc5"
server_key_version: "encryptionkey@0"
I20260810 04:10:10.824847 17680 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:10:10.843441 17680 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:10:10.845386 17680 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:10:10.845551 17680 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:10:10.845786 17680 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:10:10.846091 17680 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:10:10.846148 17680 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:10:10.846184 17680 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:10:10.846210 17680 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:10:10.852066 17680 rpc_server.cc:307] RPC server started. Bound to: 127.31.32.2:45549
I20260810 04:10:10.852200 17830 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.32.2:45549 every 8 connection(s)
I20260810 04:10:10.852536 17680 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/ts-1/data/info.pb
I20260810 04:10:10.853622 17833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:10.853228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:39943 (local address 127.31.32.2:54179)
0810 04:10:10.853380 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0810 04:10:10.853478 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.62:39943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":79,"threads_started":1}
W20260810 04:10:10.854103 17831 heartbeater.cc:644] Failed to heartbeat to 127.31.32.62:39943 (0 consecutive failures): Network error: Failed to ping master at 127.31.32.62:39943: Client connection negotiation failed: client connection to 127.31.32.62:39943: connect: Connection refused (error 111)
I20260810 04:10:10.854526 17833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:10.854365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:39943 (local address 127.31.32.2:37409)
0810 04:10:10.854468 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0810 04:10:10.854505 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.62:39943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260810 04:10:10.854818 17833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:10.854739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:39943 (local address 127.31.32.2:49923)
0810 04:10:10.854782 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0810 04:10:10.854804 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.62:39943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260810 04:10:10.854948 17831 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260810 04:10:10.856914 31872 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskws89Io/build/release/bin/kudu as pid 17680
I20260810 04:10:10.857158 31872 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskws89Io/build/release/bin/kudu
/tmp/dist-test-taskws89Io/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.32.3:44889
--local_ip_for_outbound_sockets=127.31.32.3
--tserver_master_addrs=127.31.32.62:39943
--webserver_port=43127
--webserver_interface=127.31.32.3
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.3.keytab
--principal=kudu/127.31.32.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.32.20:40835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:10:10.939898 17834 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:10:10.940093 17834 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:10:10.940133 17834 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:10:10.941288 17834 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:10:10.941331 17834 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:10:10.941356 17834 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.32.3
I20260810 04:10:10.942863 17834 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.32.20:40835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.3.keytab
--principal=kudu/127.31.32.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.32.3:44889
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.32.3
--webserver_port=43127
--tserver_master_addrs=127.31.32.62:39943
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17834
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.32.3
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:10:10.943118 17834 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:10:10.943356 17834 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:10:10.948530 17839 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:10:10.948530 17840 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:10:10.948570 17834 server_base.cc:1034] running on GCE node
W20260810 04:10:10.948565 17842 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:10:10.948994 17834 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:10:10.949245 17834 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:10:10.950403 17834 hybrid_clock.cc:648] HybridClock initialized: now 1786335010950395 us; error 45 us; skew 500 ppm
Aug 10 04:10:10 dist-test-slave-8w9v krb5kdc[16241](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335010, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:10:10.953126 17834 init.cc:377] Logged in from keytab as kudu/127.31.32.3@KRBTEST.COM (short username kudu)
I20260810 04:10:10.953563 17834 webserver.cc:460] Webserver started at http://127.31.32.3:43127/ using document root <none> and password file <none>
I20260810 04:10:10.953796 17834 fs_manager.cc:359] Metadata directory not provided
I20260810 04:10:10.953842 17834 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:10:10.955415 17834 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260810 04:10:10.956105 17852 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:10:10.956334 17834 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 04:10:10.956439 17834 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/ts-2/data,/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/ts-2/wal
uuid: "9eda5e08e6434319b8b3626b9fabf245"
format_stamp: "Formatted at 2026-08-10 04:09:29 on dist-test-slave-8w9v"
server_key: "7b9b99a18e68ebcd40b887bca207e8fd"
server_key_iv: "d591e51a46ccd8da63898fbdcd65317b"
server_key_version: "encryptionkey@0"
I20260810 04:10:10.956799 17834 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:10:10.970500 17834 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:10:10.973013 17834 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:10:10.973183 17834 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:10:10.973429 17834 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:10:10.973752 17834 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:10:10.973815 17834 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:10:10.973860 17834 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:10:10.973896 17834 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:10:10.979935 17834 rpc_server.cc:307] RPC server started. Bound to: 127.31.32.3:44889
I20260810 04:10:10.979981 17979 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.32.3:44889 every 8 connection(s)
I20260810 04:10:10.980400 17834 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786334794676589-31872-0/minicluster-data/ts-2/data/info.pb
I20260810 04:10:10.981478 17982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:10.981102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:39943 (local address 127.31.32.3:59773)
0810 04:10:10.981265 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0810 04:10:10.981337 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.62:39943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":80,"threads_started":1}
W20260810 04:10:10.981884 17980 heartbeater.cc:644] Failed to heartbeat to 127.31.32.62:39943 (0 consecutive failures): Network error: Failed to ping master at 127.31.32.62:39943: Client connection negotiation failed: client connection to 127.31.32.62:39943: connect: Connection refused (error 111)
I20260810 04:10:10.982172 17982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:10.982086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:39943 (local address 127.31.32.3:53669)
0810 04:10:10.982130 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0810 04:10:10.982155 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.62:39943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260810 04:10:10.982427 17982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:10.982379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:39943 (local address 127.31.32.3:51969)
0810 04:10:10.982399 (+ 20us) negotiation.cc:107] Waiting for socket to connect
0810 04:10:10.982416 (+ 17us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.62:39943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":12}
W20260810 04:10:10.982642 17980 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260810 04:10:10.982765 31872 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskws89Io/build/release/bin/kudu as pid 17834
I20260810 04:10:11.780968 17984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:11.780728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:39943 (local address 127.31.32.1:58525)
0810 04:10:11.780862 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0810 04:10:11.780902 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.62:39943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":73,"threads_started":1}
I20260810 04:10:11.855618 17985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:11.855308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:39943 (local address 127.31.32.2:41797)
0810 04:10:11.855459 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0810 04:10:11.855502 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.62:39943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":59,"threads_started":1}
I20260810 04:10:11.983237 17986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:11.983011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:39943 (local address 127.31.32.3:35433)
0810 04:10:11.983129 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0810 04:10:11.983176 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.62:39943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":60,"threads_started":1}
I20260810 04:10:12.781714 17987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:12.781458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:39943 (local address 127.31.32.1:47797)
0810 04:10:12.781628 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0810 04:10:12.781666 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.62:39943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":78,"threads_started":1}
I20260810 04:10:12.856287 17988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:12.856076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:39943 (local address 127.31.32.2:57981)
0810 04:10:12.856214 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0810 04:10:12.856247 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.62:39943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":71,"threads_started":1}
I20260810 04:10:12.983920 17989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:12.983706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:39943 (local address 127.31.32.3:43299)
0810 04:10:12.983842 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0810 04:10:12.983878 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.62:39943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":66,"threads_started":1}
I20260810 04:10:13.782505 17990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:13.782215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:39943 (local address 127.31.32.1:60727)
0810 04:10:13.782375 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0810 04:10:13.782425 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.62:39943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":103,"threads_started":1}
I20260810 04:10:13.856986 17991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:13.856745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:39943 (local address 127.31.32.2:59487)
0810 04:10:13.856899 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0810 04:10:13.856939 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.62:39943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":67,"threads_started":1}
I20260810 04:10:13.984534 17992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:13.984304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:39943 (local address 127.31.32.3:38493)
0810 04:10:13.984441 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0810 04:10:13.984474 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.62:39943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":67,"threads_started":1}
I20260810 04:10:14.783171 17993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:14.782965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:39943 (local address 127.31.32.1:50921)
0810 04:10:14.783091 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0810 04:10:14.783123 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.62:39943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":68,"threads_started":1}
I20260810 04:10:14.857761 17994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:14.857550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:39943 (local address 127.31.32.2:44075)
0810 04:10:14.857675 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0810 04:10:14.857719 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.62:39943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":62,"threads_started":1}
I20260810 04:10:14.985221 17995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:14.985003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:39943 (local address 127.31.32.3:51695)
0810 04:10:14.985143 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0810 04:10:14.985179 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.62:39943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":89,"threads_started":1}
I20260810 04:10:15.783964 17996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:15.783728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:39943 (local address 127.31.32.1:34763)
0810 04:10:15.783871 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0810 04:10:15.783916 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.62:39943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":67,"threads_started":1}
I20260810 04:10:15.858460 17997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:15.858205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:39943 (local address 127.31.32.2:51345)
0810 04:10:15.858380 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0810 04:10:15.858417 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.62:39943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":96,"threads_started":1}
I20260810 04:10:15.985918 17998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:15.985679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:39943 (local address 127.31.32.3:46859)
0810 04:10:15.985828 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0810 04:10:15.985872 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.62:39943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":69,"threads_started":1}
I20260810 04:10:16.784883 17999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:16.784543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:39943 (local address 127.31.32.1:60943)
0810 04:10:16.784776 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0810 04:10:16.784820 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.62:39943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":98,"threads_started":1}
I20260810 04:10:16.859165 18000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:16.858916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:39943 (local address 127.31.32.2:41445)
0810 04:10:16.859085 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0810 04:10:16.859123 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.62:39943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":76,"threads_started":1}
I20260810 04:10:16.986670 18001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:16.986379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:39943 (local address 127.31.32.3:43867)
0810 04:10:16.986590 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0810 04:10:16.986626 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.62:39943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":133,"threads_started":1}
I20260810 04:10:17.785755 18002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:17.785492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:39943 (local address 127.31.32.1:33671)
0810 04:10:17.785669 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0810 04:10:17.785707 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.62:39943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":82,"threads_started":1}
I20260810 04:10:17.859910 18003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:17.859690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:39943 (local address 127.31.32.2:33333)
0810 04:10:17.859819 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0810 04:10:17.859860 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.62:39943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":62,"threads_started":1}
I20260810 04:10:17.987372 18005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:17.987142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:39943 (local address 127.31.32.3:36995)
0810 04:10:17.987278 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0810 04:10:17.987317 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.62:39943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":65,"threads_started":1}
I20260810 04:10:18.786459 18006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:18.786247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:39943 (local address 127.31.32.1:59311)
0810 04:10:18.786378 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0810 04:10:18.786411 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.62:39943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":67,"threads_started":1}
I20260810 04:10:18.860678 18008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:18.860397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:39943 (local address 127.31.32.2:43887)
0810 04:10:18.860587 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0810 04:10:18.860629 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.62:39943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":89,"threads_started":1}
I20260810 04:10:18.988090 18009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:18.987878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:39943 (local address 127.31.32.3:42767)
0810 04:10:18.988000 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0810 04:10:18.988041 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.62:39943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":61,"threads_started":1}
I20260810 04:10:19.787213 18011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:19.786993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:39943 (local address 127.31.32.1:52279)
0810 04:10:19.787125 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0810 04:10:19.787165 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.62:39943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":86,"threads_started":1}
I20260810 04:10:19.861486 18012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:19.861225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:39943 (local address 127.31.32.2:40285)
0810 04:10:19.861401 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0810 04:10:19.861436 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.62:39943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":104,"threads_started":1}
I20260810 04:10:19.988803 18013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:19.988575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:39943 (local address 127.31.32.3:47013)
0810 04:10:19.988717 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0810 04:10:19.988754 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.62:39943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":63,"threads_started":1}
I20260810 04:10:20.787930 18014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:20.787713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:39943 (local address 127.31.32.1:40337)
0810 04:10:20.787843 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0810 04:10:20.787879 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.62:39943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":61,"threads_started":1}
I20260810 04:10:20.862139 18015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:20.861937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:39943 (local address 127.31.32.2:36553)
0810 04:10:20.862059 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0810 04:10:20.862092 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.62:39943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":61,"threads_started":1}
I20260810 04:10:20.989519 18016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:20.989290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:39943 (local address 127.31.32.3:53119)
0810 04:10:20.989436 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0810 04:10:20.989474 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.62:39943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":82,"threads_started":1}
I20260810 04:10:21.788635 18017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:21.788368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:39943 (local address 127.31.32.1:55577)
0810 04:10:21.788531 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0810 04:10:21.788567 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.62:39943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":91,"threads_started":1}
I20260810 04:10:21.862795 18018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:21.862568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:39943 (local address 127.31.32.2:48281)
0810 04:10:21.862709 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0810 04:10:21.862746 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.62:39943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":63,"threads_started":1}
I20260810 04:10:21.990149 18019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:21.989939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:39943 (local address 127.31.32.3:41541)
0810 04:10:21.990075 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0810 04:10:21.990106 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.62:39943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":78,"threads_started":1}
I20260810 04:10:22.789300 18020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:22.789083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:39943 (local address 127.31.32.1:48695)
0810 04:10:22.789214 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0810 04:10:22.789251 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.62:39943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":63,"threads_started":1}
I20260810 04:10:22.863492 18021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:22.863263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:39943 (local address 127.31.32.2:60369)
0810 04:10:22.863408 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0810 04:10:22.863445 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.62:39943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":66,"threads_started":1}
I20260810 04:10:22.990795 18023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:22.990587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:39943 (local address 127.31.32.3:40561)
0810 04:10:22.990712 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0810 04:10:22.990748 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.62:39943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":63,"threads_started":1}
I20260810 04:10:23.789940 18024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:23.789729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:39943 (local address 127.31.32.1:51087)
0810 04:10:23.789853 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0810 04:10:23.789891 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.62:39943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":65,"threads_started":1}
I20260810 04:10:23.864255 18026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:23.863985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:39943 (local address 127.31.32.2:54727)
0810 04:10:23.864161 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0810 04:10:23.864206 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.62:39943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":65,"threads_started":1}
I20260810 04:10:23.991467 18027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:23.991240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:39943 (local address 127.31.32.3:58617)
0810 04:10:23.991385 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0810 04:10:23.991419 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.62:39943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":64,"threads_started":1}
I20260810 04:10:24.790659 18030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:24.790410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:39943 (local address 127.31.32.1:60041)
0810 04:10:24.790558 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0810 04:10:24.790595 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.62:39943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":79,"threads_started":1}
I20260810 04:10:24.864974 18033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:24.864751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:39943 (local address 127.31.32.2:40943)
0810 04:10:24.864893 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0810 04:10:24.864929 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.62:39943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":79,"threads_started":1}
I20260810 04:10:24.992386 18034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:24.992066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:39943 (local address 127.31.32.3:34191)
0810 04:10:24.992282 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0810 04:10:24.992333 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.62:39943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":139,"threads_started":1}
I20260810 04:10:25.791396 18035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:25.791193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:39943 (local address 127.31.32.1:36783)
0810 04:10:25.791312 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0810 04:10:25.791348 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.62:39943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":74,"threads_started":1}
I20260810 04:10:25.865715 18036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:25.865520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:39943 (local address 127.31.32.2:34017)
0810 04:10:25.865638 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0810 04:10:25.865670 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.62:39943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":63,"threads_started":1}
I20260810 04:10:25.989392 31872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskws89Io/build/release/bin/kudu with pid 17488
I20260810 04:10:25.993109 18039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:25.992908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:39943 (local address 127.31.32.3:55611)
0810 04:10:25.993034 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0810 04:10:25.993066 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.62:39943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":76,"threads_started":1}
I20260810 04:10:25.996716 31872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskws89Io/build/release/bin/kudu with pid 17680
I20260810 04:10:26.003810 31872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskws89Io/build/release/bin/kudu with pid 17834
2026-08-10T04:10:26Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58510 ms)
[----------] 30 tests from SecurityITest (231326 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786334794676589-31872-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 04:10:26 dist-test-slave-8w9v krb5kdc[18042](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 04:10:26 dist-test-slave-8w9v krb5kdc[18042](info): set up 2 sockets
Aug 10 04:10:26 dist-test-slave-8w9v krb5kdc[18042](info): commencing operation
krb5kdc: starting...
W20260810 04:10:27.098806 31872 mini_kdc.cc:121] Time spent starting KDC: real 1.072s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786334794676589-31872-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786334794676589-31872-0/krb5kdc/test-user.keytab.
Aug 10 04:10:27 dist-test-slave-8w9v krb5kdc[18042](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335027, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T04:10:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T04:10:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.32.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.32.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.32.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.62.keytab.
Entry for principal kudu/127.31.32.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.62.keytab.
I20260810 04:10:27.222782 31872 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskws89Io/build/release/bin/kudu
/tmp/dist-test-taskws89Io/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786334794676589-31872-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786334794676589-31872-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786334794676589-31872-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786334794676589-31872-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.32.62:46103
--webserver_interface=127.31.32.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.62.keytab
--principal=kudu/127.31.32.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.32.20:34757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.32.62:46103
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786334794676589-31872-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786334794676589-31872-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786334794676589-31872-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:10:27.307011 18058 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:10:27.307288 18058 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:10:27.307333 18058 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:10:27.308537 18058 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:10:27.308635 18058 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:10:27.308663 18058 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:10:27.308696 18058 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:10:27.308732 18058 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:10:27.310232 18058 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.32.20:34757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786334794676589-31872-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786334794676589-31872-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.32.62:46103
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.62.keytab
--principal=kudu/127.31.32.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.32.62:46103
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786334794676589-31872-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.32.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18058
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786334794676589-31872-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:10:27.310537 18058 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:10:27.310799 18058 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:10:27.316236 18064 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:10:27.316314 18066 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:10:27.316368 18058 server_base.cc:1034] running on GCE node
W20260810 04:10:27.316277 18063 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:10:27.316747 18058 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:10:27.317003 18058 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:10:27.318169 18058 hybrid_clock.cc:648] HybridClock initialized: now 1786335027318141 us; error 37 us; skew 500 ppm
Aug 10 04:10:27 dist-test-slave-8w9v krb5kdc[18042](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335027, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:10:27.320803 18058 init.cc:377] Logged in from keytab as kudu/127.31.32.62@KRBTEST.COM (short username kudu)
I20260810 04:10:27.321305 18058 webserver.cc:460] Webserver started at http://127.31.32.62:42089/ using document root <none> and password file <none>
I20260810 04:10:27.321543 18058 fs_manager.cc:359] Metadata directory not provided
I20260810 04:10:27.321619 18058 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:10:27.321787 18058 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:10:27.323136 18058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786334794676589-31872-0/minicluster-data/master-0/data/instance:
uuid: "9830aca959a14e07baa7f8368a283bbb"
format_stamp: "Formatted at 2026-08-10 04:10:27 on dist-test-slave-8w9v"
server_key: "a4af9fdc117e5a910d642ebf26938a8f"
server_key_iv: "85812f649438ec50cda9113c093e2b73"
server_key_version: "encryptionkey@0"
I20260810 04:10:27.323604 18058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786334794676589-31872-0/minicluster-data/master-0/wal/instance:
uuid: "9830aca959a14e07baa7f8368a283bbb"
format_stamp: "Formatted at 2026-08-10 04:10:27 on dist-test-slave-8w9v"
server_key: "a4af9fdc117e5a910d642ebf26938a8f"
server_key_iv: "85812f649438ec50cda9113c093e2b73"
server_key_version: "encryptionkey@0"
I20260810 04:10:27.325769 18058 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.004s
I20260810 04:10:27.326689 18073 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:10:27.326859 18058 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 04:10:27.326975 18058 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786334794676589-31872-0/minicluster-data/master-0/data,/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786334794676589-31872-0/minicluster-data/master-0/wal
uuid: "9830aca959a14e07baa7f8368a283bbb"
format_stamp: "Formatted at 2026-08-10 04:10:27 on dist-test-slave-8w9v"
server_key: "a4af9fdc117e5a910d642ebf26938a8f"
server_key_iv: "85812f649438ec50cda9113c093e2b73"
server_key_version: "encryptionkey@0"
I20260810 04:10:27.327070 18058 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786334794676589-31872-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786334794676589-31872-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786334794676589-31872-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:10:27.339567 18058 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:10:27.341516 18058 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:10:27.341696 18058 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:10:27.346201 18058 rpc_server.cc:307] RPC server started. Bound to: 127.31.32.62:46103
I20260810 04:10:27.346244 18132 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.32.62:46103 every 8 connection(s)
I20260810 04:10:27.346604 18058 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786334794676589-31872-0/minicluster-data/master-0/data/info.pb
I20260810 04:10:27.347476 18133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:10:27.347947 31872 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskws89Io/build/release/bin/kudu as pid 18058
I20260810 04:10:27.348073 31872 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786334794676589-31872-0/minicluster-data/master-0/wal/instance
I20260810 04:10:27.348304 31872 external_mini_cluster.cc:1428] Setting key 8e85b5f63b5470bb274e04950cb9a0a5
I20260810 04:10:27.350622 18133 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9830aca959a14e07baa7f8368a283bbb: Bootstrap starting.
I20260810 04:10:27.351217 18133 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9830aca959a14e07baa7f8368a283bbb: Neither blocks nor log segments found. Creating new log.
I20260810 04:10:27.351506 18133 log.cc:824] T 00000000000000000000000000000000 P 9830aca959a14e07baa7f8368a283bbb: Log is configured to *not* fsync() on all Append() calls
I20260810 04:10:27.352267 18133 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9830aca959a14e07baa7f8368a283bbb: No bootstrap required, opened a new log
I20260810 04:10:27.354877 18133 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9830aca959a14e07baa7f8368a283bbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9830aca959a14e07baa7f8368a283bbb" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 46103 } }
I20260810 04:10:27.355033 18133 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9830aca959a14e07baa7f8368a283bbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:10:27.355057 18133 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9830aca959a14e07baa7f8368a283bbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9830aca959a14e07baa7f8368a283bbb, State: Initialized, Role: FOLLOWER
I20260810 04:10:27.355193 18133 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9830aca959a14e07baa7f8368a283bbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9830aca959a14e07baa7f8368a283bbb" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 46103 } }
I20260810 04:10:27.355270 18133 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9830aca959a14e07baa7f8368a283bbb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 04:10:27.355293 18133 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9830aca959a14e07baa7f8368a283bbb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 04:10:27.355355 18133 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9830aca959a14e07baa7f8368a283bbb [term 0 FOLLOWER]: Advancing to term 1
Aug 10 04:10:27 dist-test-slave-8w9v krb5kdc[18042](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335027, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.32.62@KRBTEST.COM
I20260810 04:10:27.355964 18133 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9830aca959a14e07baa7f8368a283bbb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9830aca959a14e07baa7f8368a283bbb" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 46103 } }
I20260810 04:10:27.356104 18133 leader_election.cc:302] T 00000000000000000000000000000000 P 9830aca959a14e07baa7f8368a283bbb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9830aca959a14e07baa7f8368a283bbb; no voters:
I20260810 04:10:27.356391 18133 leader_election.cc:288] T 00000000000000000000000000000000 P 9830aca959a14e07baa7f8368a283bbb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 04:10:27.356433 18138 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9830aca959a14e07baa7f8368a283bbb [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:10:27.356691 18133 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9830aca959a14e07baa7f8368a283bbb [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:10:27.356564 18138 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9830aca959a14e07baa7f8368a283bbb [term 1 LEADER]: Becoming Leader. State: Replica: 9830aca959a14e07baa7f8368a283bbb, State: Running, Role: LEADER
I20260810 04:10:27.358044 18138 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9830aca959a14e07baa7f8368a283bbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9830aca959a14e07baa7f8368a283bbb" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 46103 } }
I20260810 04:10:27.358577 18141 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9830aca959a14e07baa7f8368a283bbb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9830aca959a14e07baa7f8368a283bbb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9830aca959a14e07baa7f8368a283bbb" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 46103 } } }
I20260810 04:10:27.358665 18141 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9830aca959a14e07baa7f8368a283bbb [sys.catalog]: This master's current role is: LEADER
I20260810 04:10:27.358754 18139 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9830aca959a14e07baa7f8368a283bbb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9830aca959a14e07baa7f8368a283bbb. Latest consensus state: current_term: 1 leader_uuid: "9830aca959a14e07baa7f8368a283bbb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9830aca959a14e07baa7f8368a283bbb" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 46103 } } }
I20260810 04:10:27.358861 18139 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9830aca959a14e07baa7f8368a283bbb [sys.catalog]: This master's current role is: LEADER
I20260810 04:10:27.359409 18156 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 04:10:27.359839 18156 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 04:10:27.359849 18136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:27.348898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39002 (local address 127.31.32.62:46103)
0810 04:10:27.349088 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0810 04:10:27.349108 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0810 04:10:27.349135 (+ 27us) server_negotiation.cc:407] Connection header received
0810 04:10:27.349547 (+ 412us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:10:27.349553 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:10:27.349599 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:10:27.349722 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:10:27.350526 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:27.351521 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:27.352589 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:27.352991 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:27.353086 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:10:27.356089 (+ 3003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:10:27.356093 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:10:27.356094 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:10:27.356106 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:10:27.358012 (+ 1906us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:10:27.358435 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:10:27.358438 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:10:27.358439 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:10:27.358561 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:10:27.358823 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:10:27.358824 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:10:27.358825 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:10:27.359143 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:10:27.359317 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0810 04:10:27.359472 (+ 155us) server_negotiation.cc:299] Negotiation successful
0810 04:10:27.359615 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":88,"threads_started":1}
I20260810 04:10:27.361519 18156 catalog_manager.cc:1347] Generated new cluster ID: 746514e5c8be40819d4848dcc0625814
I20260810 04:10:27.361589 18156 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 04:10:27.366083 18156 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 04:10:27.366551 18156 catalog_manager.cc:1504] Loading token signing keys...
I20260810 04:10:27.369742 18156 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9830aca959a14e07baa7f8368a283bbb: Generated new TSK 0
I20260810 04:10:27.369922 18156 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.32.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.32.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.32.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.1.keytab.
Entry for principal kudu/127.31.32.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.1.keytab.
I20260810 04:10:27.403299 31872 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskws89Io/build/release/bin/kudu
/tmp/dist-test-taskws89Io/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786334794676589-31872-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786334794676589-31872-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786334794676589-31872-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786334794676589-31872-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.32.1:0
--local_ip_for_outbound_sockets=127.31.32.1
--webserver_interface=127.31.32.1
--webserver_port=0
--tserver_master_addrs=127.31.32.62:46103
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.1.keytab
--principal=kudu/127.31.32.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.32.20:34757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786334794676589-31872-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786334794676589-31872-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786334794676589-31872-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:10:27.500809 18162 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:10:27.501046 18162 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:10:27.501083 18162 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:10:27.502242 18162 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:10:27.502314 18162 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:10:27.502358 18162 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.32.1
I20260810 04:10:27.503836 18162 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.32.20:34757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786334794676589-31872-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786334794676589-31872-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.1.keytab
--principal=kudu/127.31.32.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.32.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786334794676589-31872-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.32.1
--webserver_port=0
--tserver_master_addrs=127.31.32.62:46103
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18162
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.32.1
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786334794676589-31872-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:10:27.504165 18162 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:10:27.504426 18162 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 04:10:27.510525 18162 server_base.cc:1034] running on GCE node
W20260810 04:10:27.510492 18170 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:10:27.510670 18169 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:10:27.510506 18172 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:10:27.510977 18162 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:10:27.511276 18162 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:10:27.512439 18162 hybrid_clock.cc:648] HybridClock initialized: now 1786335027512423 us; error 32 us; skew 500 ppm
Aug 10 04:10:27 dist-test-slave-8w9v krb5kdc[18042](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335027, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:10:27.515054 18162 init.cc:377] Logged in from keytab as kudu/127.31.32.1@KRBTEST.COM (short username kudu)
I20260810 04:10:27.515539 18162 webserver.cc:460] Webserver started at http://127.31.32.1:36261/ using document root <none> and password file <none>
I20260810 04:10:27.515779 18162 fs_manager.cc:359] Metadata directory not provided
I20260810 04:10:27.515822 18162 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:10:27.515913 18162 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:10:27.517041 18162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786334794676589-31872-0/minicluster-data/ts-0/data/instance:
uuid: "3316ed4624f2494d994309a8b80fe5cc"
format_stamp: "Formatted at 2026-08-10 04:10:27 on dist-test-slave-8w9v"
server_key: "47e86070f5e2ebe965e072047e74d68d"
server_key_iv: "091cc07e74a981f811f317902d449b06"
server_key_version: "encryptionkey@0"
I20260810 04:10:27.517356 18162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786334794676589-31872-0/minicluster-data/ts-0/wal/instance:
uuid: "3316ed4624f2494d994309a8b80fe5cc"
format_stamp: "Formatted at 2026-08-10 04:10:27 on dist-test-slave-8w9v"
server_key: "47e86070f5e2ebe965e072047e74d68d"
server_key_iv: "091cc07e74a981f811f317902d449b06"
server_key_version: "encryptionkey@0"
I20260810 04:10:27.518692 18162 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260810 04:10:27.519451 18181 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:10:27.519649 18162 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 04:10:27.519768 18162 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786334794676589-31872-0/minicluster-data/ts-0/data,/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786334794676589-31872-0/minicluster-data/ts-0/wal
uuid: "3316ed4624f2494d994309a8b80fe5cc"
format_stamp: "Formatted at 2026-08-10 04:10:27 on dist-test-slave-8w9v"
server_key: "47e86070f5e2ebe965e072047e74d68d"
server_key_iv: "091cc07e74a981f811f317902d449b06"
server_key_version: "encryptionkey@0"
I20260810 04:10:27.519846 18162 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786334794676589-31872-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786334794676589-31872-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786334794676589-31872-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:10:27.530952 18162 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:10:27.533389 18162 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:10:27.533550 18162 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:10:27.533790 18162 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:10:27.534111 18162 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:10:27.534150 18162 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:10:27.534192 18162 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:10:27.534226 18162 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:10:27.540305 18162 rpc_server.cc:307] RPC server started. Bound to: 127.31.32.1:44385
I20260810 04:10:27.540400 18307 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.32.1:44385 every 8 connection(s)
I20260810 04:10:27.540723 18162 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786334794676589-31872-0/minicluster-data/ts-0/data/info.pb
Aug 10 04:10:27 dist-test-slave-8w9v krb5kdc[18042](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335027, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.1@KRBTEST.COM for kudu/127.31.32.62@KRBTEST.COM
I20260810 04:10:27.549873 18136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:27.541570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.1:42515 (local address 127.31.32.62:46103)
0810 04:10:27.541621 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0810 04:10:27.541623 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:10:27.541833 (+ 210us) server_negotiation.cc:407] Connection header received
0810 04:10:27.542281 (+ 448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:10:27.542282 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:10:27.542294 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:10:27.542380 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:10:27.543350 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:27.543846 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:27.544823 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:27.545042 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:27.545098 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:10:27.547210 (+ 2112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:10:27.547224 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:10:27.547224 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:10:27.547230 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:10:27.548712 (+ 1482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:10:27.549059 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:10:27.549060 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:10:27.549060 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:10:27.549111 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:10:27.549358 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:10:27.549359 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:10:27.549359 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:10:27.549456 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:10:27.549506 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0810 04:10:27.549734 (+ 228us) server_negotiation.cc:299] Negotiation successful
0810 04:10:27.549803 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260810 04:10:27.550071 18311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:27.541479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:46103 (local address 127.31.32.1:42515)
0810 04:10:27.541659 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0810 04:10:27.541703 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 04:10:27.542195 (+ 492us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:10:27.542458 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:10:27.542462 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:10:27.543058 (+ 596us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:10:27.543257 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:10:27.543261 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:10:27.543928 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:10:27.543929 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:10:27.544739 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:10:27.544741 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:10:27.545118 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:10:27.545119 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:10:27.545222 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:10:27.545711 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:10:27.545719 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:10:27.547106 (+ 1387us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:10:27.548780 (+ 1674us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:10:27.548786 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:10:27.548788 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:10:27.548993 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:10:27.549170 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:10:27.549170 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:10:27.549170 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:10:27.549284 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:10:27.549512 (+ 228us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:10:27.549517 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:10:27.549628 (+ 111us) client_negotiation.cc:769] Sending connection context
0810 04:10:27.549708 (+ 80us) client_negotiation.cc:240] Negotiation successful
0810 04:10:27.549824 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":102,"threads_started":1}
I20260810 04:10:27.550243 31872 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskws89Io/build/release/bin/kudu as pid 18162
I20260810 04:10:27.550406 31872 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786334794676589-31872-0/minicluster-data/ts-0/wal/instance
I20260810 04:10:27.550621 18308 heartbeater.cc:347] Connected to a master server at 127.31.32.62:46103
I20260810 04:10:27.550733 18308 heartbeater.cc:464] Registering TS with master...
I20260810 04:10:27.550727 31872 external_mini_cluster.cc:1428] Setting key 6dc24a5adfc8c1c34fca582e545efca7
I20260810 04:10:27.550959 18308 heartbeater.cc:511] Master 127.31.32.62:46103 requested a full tablet report, sending...
I20260810 04:10:27.551556 18091 ts_manager.cc:194] Registered new tserver with Master: 3316ed4624f2494d994309a8b80fe5cc (127.31.32.1:44385)
I20260810 04:10:27.552474 18091 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.32.1@KRBTEST.COM'} at 127.31.32.1:42515
WARNING: no policy specified for kudu/127.31.32.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.32.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.32.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.2.keytab.
Entry for principal kudu/127.31.32.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.2.keytab.
I20260810 04:10:27.577346 31872 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskws89Io/build/release/bin/kudu
/tmp/dist-test-taskws89Io/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786334794676589-31872-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786334794676589-31872-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786334794676589-31872-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786334794676589-31872-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.32.2:0
--local_ip_for_outbound_sockets=127.31.32.2
--webserver_interface=127.31.32.2
--webserver_port=0
--tserver_master_addrs=127.31.32.62:46103
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.2.keytab
--principal=kudu/127.31.32.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.32.20:34757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786334794676589-31872-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786334794676589-31872-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786334794676589-31872-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:10:27.662236 18315 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:10:27.662490 18315 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:10:27.662533 18315 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:10:27.663738 18315 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:10:27.663808 18315 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:10:27.663854 18315 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.32.2
I20260810 04:10:27.665349 18315 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.32.20:34757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786334794676589-31872-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786334794676589-31872-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.2.keytab
--principal=kudu/127.31.32.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.32.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786334794676589-31872-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.32.2
--webserver_port=0
--tserver_master_addrs=127.31.32.62:46103
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18315
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.32.2
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786334794676589-31872-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:10:27.665645 18315 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:10:27.665884 18315 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:10:27.671306 18323 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:10:27.671276 18321 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:10:27.671288 18320 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:10:27.671322 18315 server_base.cc:1034] running on GCE node
I20260810 04:10:27.671729 18315 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:10:27.671969 18315 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:10:27.673132 18315 hybrid_clock.cc:648] HybridClock initialized: now 1786335027673129 us; error 51 us; skew 500 ppm
Aug 10 04:10:27 dist-test-slave-8w9v krb5kdc[18042](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335027, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:10:27.675721 18315 init.cc:377] Logged in from keytab as kudu/127.31.32.2@KRBTEST.COM (short username kudu)
I20260810 04:10:27.676324 18315 webserver.cc:460] Webserver started at http://127.31.32.2:33639/ using document root <none> and password file <none>
I20260810 04:10:27.676576 18315 fs_manager.cc:359] Metadata directory not provided
I20260810 04:10:27.676647 18315 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:10:27.676815 18315 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:10:27.677896 18315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786334794676589-31872-0/minicluster-data/ts-1/data/instance:
uuid: "6e2153c35afe4e3f9eec16136f1220ec"
format_stamp: "Formatted at 2026-08-10 04:10:27 on dist-test-slave-8w9v"
server_key: "c5b8a06e62030a6005f3f504c8421909"
server_key_iv: "173c539898440b6a64eb0dc01dcb1749"
server_key_version: "encryptionkey@0"
I20260810 04:10:27.678277 18315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786334794676589-31872-0/minicluster-data/ts-1/wal/instance:
uuid: "6e2153c35afe4e3f9eec16136f1220ec"
format_stamp: "Formatted at 2026-08-10 04:10:27 on dist-test-slave-8w9v"
server_key: "c5b8a06e62030a6005f3f504c8421909"
server_key_iv: "173c539898440b6a64eb0dc01dcb1749"
server_key_version: "encryptionkey@0"
I20260810 04:10:27.679690 18315 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 04:10:27.680533 18331 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:10:27.680694 18315 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260810 04:10:27.680805 18315 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786334794676589-31872-0/minicluster-data/ts-1/data,/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786334794676589-31872-0/minicluster-data/ts-1/wal
uuid: "6e2153c35afe4e3f9eec16136f1220ec"
format_stamp: "Formatted at 2026-08-10 04:10:27 on dist-test-slave-8w9v"
server_key: "c5b8a06e62030a6005f3f504c8421909"
server_key_iv: "173c539898440b6a64eb0dc01dcb1749"
server_key_version: "encryptionkey@0"
I20260810 04:10:27.680881 18315 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786334794676589-31872-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786334794676589-31872-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786334794676589-31872-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:10:27.692363 18315 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:10:27.694468 18315 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:10:27.694646 18315 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:10:27.694887 18315 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:10:27.695253 18315 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:10:27.695319 18315 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:10:27.695361 18315 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:10:27.695390 18315 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:10:27.701464 18315 rpc_server.cc:307] RPC server started. Bound to: 127.31.32.2:38617
I20260810 04:10:27.701516 18456 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.32.2:38617 every 8 connection(s)
I20260810 04:10:27.701844 18315 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786334794676589-31872-0/minicluster-data/ts-1/data/info.pb
I20260810 04:10:27.703125 31872 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskws89Io/build/release/bin/kudu as pid 18315
I20260810 04:10:27.703238 31872 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786334794676589-31872-0/minicluster-data/ts-1/wal/instance
I20260810 04:10:27.703461 31872 external_mini_cluster.cc:1428] Setting key ef928a444829204a2fd9df2ee2683323
Aug 10 04:10:27 dist-test-slave-8w9v krb5kdc[18042](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335027, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.2@KRBTEST.COM for kudu/127.31.32.62@KRBTEST.COM
I20260810 04:10:27.712597 18136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:27.702621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.2:56093 (local address 127.31.32.62:46103)
0810 04:10:27.702701 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0810 04:10:27.702703 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:10:27.702935 (+ 232us) server_negotiation.cc:407] Connection header received
0810 04:10:27.703689 (+ 754us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:10:27.703691 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:10:27.703704 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:10:27.703769 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:10:27.705134 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:27.705627 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:27.706766 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:27.707077 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:27.707157 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:10:27.709639 (+ 2482us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:10:27.709641 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:10:27.709641 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:10:27.709650 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:10:27.711089 (+ 1439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:10:27.711534 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:10:27.711536 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:10:27.711536 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:10:27.711601 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:10:27.711886 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:10:27.711887 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:10:27.711887 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:10:27.712006 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:10:27.712081 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0810 04:10:27.712407 (+ 326us) server_negotiation.cc:299] Negotiation successful
0810 04:10:27.712504 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
WARNING: no policy specified for kudu/127.31.32.3@KRBTEST.COM; defaulting to no policy
I20260810 04:10:27.712798 18459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:27.702565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:46103 (local address 127.31.32.2:56093)
0810 04:10:27.702760 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0810 04:10:27.702815 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0810 04:10:27.703568 (+ 753us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:10:27.703864 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:10:27.703876 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:10:27.704747 (+ 871us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:10:27.705031 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:10:27.705038 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:10:27.705720 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:10:27.705721 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:10:27.706674 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:10:27.706677 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:10:27.707181 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:10:27.707182 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:10:27.707357 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:10:27.707868 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:10:27.707879 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:10:27.709536 (+ 1657us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:10:27.711194 (+ 1658us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:10:27.711201 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:10:27.711204 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:10:27.711452 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:10:27.711673 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:10:27.711674 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:10:27.711675 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:10:27.711808 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:10:27.712073 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:10:27.712081 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:10:27.712273 (+ 192us) client_negotiation.cc:769] Sending connection context
0810 04:10:27.712384 (+ 111us) client_negotiation.cc:240] Negotiation successful
0810 04:10:27.712550 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":92,"threads_started":1}
I20260810 04:10:27.713478 18457 heartbeater.cc:347] Connected to a master server at 127.31.32.62:46103
I20260810 04:10:27.713608 18457 heartbeater.cc:464] Registering TS with master...
I20260810 04:10:27.713891 18457 heartbeater.cc:511] Master 127.31.32.62:46103 requested a full tablet report, sending...
I20260810 04:10:27.714285 18091 ts_manager.cc:194] Registered new tserver with Master: 6e2153c35afe4e3f9eec16136f1220ec (127.31.32.2:38617)
I20260810 04:10:27.714668 18091 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.32.2@KRBTEST.COM'} at 127.31.32.2:56093
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.32.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.32.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.3.keytab.
Entry for principal kudu/127.31.32.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.3.keytab.
I20260810 04:10:27.730531 31872 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskws89Io/build/release/bin/kudu
/tmp/dist-test-taskws89Io/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786334794676589-31872-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786334794676589-31872-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786334794676589-31872-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786334794676589-31872-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.32.3:0
--local_ip_for_outbound_sockets=127.31.32.3
--webserver_interface=127.31.32.3
--webserver_port=0
--tserver_master_addrs=127.31.32.62:46103
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.3.keytab
--principal=kudu/127.31.32.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.32.20:34757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786334794676589-31872-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786334794676589-31872-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786334794676589-31872-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:10:27.826627 18462 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:10:27.826840 18462 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:10:27.826864 18462 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:10:27.827927 18462 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:10:27.827962 18462 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:10:27.827987 18462 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.32.3
I20260810 04:10:27.829382 18462 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.32.20:34757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786334794676589-31872-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786334794676589-31872-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.3.keytab
--principal=kudu/127.31.32.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.32.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786334794676589-31872-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.32.3
--webserver_port=0
--tserver_master_addrs=127.31.32.62:46103
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18462
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.32.3
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786334794676589-31872-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:10:27.829684 18462 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:10:27.829927 18462 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:10:27.835099 18471 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:10:27.835098 18468 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:10:27.835182 18467 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:10:27.835395 18462 server_base.cc:1034] running on GCE node
I20260810 04:10:27.835620 18462 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:10:27.835920 18462 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:10:27.837139 18462 hybrid_clock.cc:648] HybridClock initialized: now 1786335027837120 us; error 55 us; skew 500 ppm
Aug 10 04:10:27 dist-test-slave-8w9v krb5kdc[18042](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335027, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:10:27.839669 18462 init.cc:377] Logged in from keytab as kudu/127.31.32.3@KRBTEST.COM (short username kudu)
I20260810 04:10:27.840209 18462 webserver.cc:460] Webserver started at http://127.31.32.3:33493/ using document root <none> and password file <none>
I20260810 04:10:27.840445 18462 fs_manager.cc:359] Metadata directory not provided
I20260810 04:10:27.840513 18462 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:10:27.840651 18462 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:10:27.841742 18462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786334794676589-31872-0/minicluster-data/ts-2/data/instance:
uuid: "cc4f95cac7284b85977f81deb3765372"
format_stamp: "Formatted at 2026-08-10 04:10:27 on dist-test-slave-8w9v"
server_key: "df73c772907b08726a0f9b764d098736"
server_key_iv: "31b7305cbc76c0036defc71f30f560f1"
server_key_version: "encryptionkey@0"
I20260810 04:10:27.842127 18462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786334794676589-31872-0/minicluster-data/ts-2/wal/instance:
uuid: "cc4f95cac7284b85977f81deb3765372"
format_stamp: "Formatted at 2026-08-10 04:10:27 on dist-test-slave-8w9v"
server_key: "df73c772907b08726a0f9b764d098736"
server_key_iv: "31b7305cbc76c0036defc71f30f560f1"
server_key_version: "encryptionkey@0"
I20260810 04:10:27.843523 18462 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 04:10:27.844357 18479 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:10:27.844539 18462 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 04:10:27.844645 18462 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786334794676589-31872-0/minicluster-data/ts-2/data,/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786334794676589-31872-0/minicluster-data/ts-2/wal
uuid: "cc4f95cac7284b85977f81deb3765372"
format_stamp: "Formatted at 2026-08-10 04:10:27 on dist-test-slave-8w9v"
server_key: "df73c772907b08726a0f9b764d098736"
server_key_iv: "31b7305cbc76c0036defc71f30f560f1"
server_key_version: "encryptionkey@0"
I20260810 04:10:27.844731 18462 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786334794676589-31872-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786334794676589-31872-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786334794676589-31872-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:10:27.859956 18462 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:10:27.862102 18462 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:10:27.862231 18462 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:10:27.862433 18462 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:10:27.862704 18462 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:10:27.862747 18462 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:10:27.862774 18462 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:10:27.862789 18462 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:10:27.869426 18462 rpc_server.cc:307] RPC server started. Bound to: 127.31.32.3:35761
I20260810 04:10:27.869484 18607 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.32.3:35761 every 8 connection(s)
I20260810 04:10:27.870003 18462 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786334794676589-31872-0/minicluster-data/ts-2/data/info.pb
I20260810 04:10:27.876211 31872 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskws89Io/build/release/bin/kudu as pid 18462
I20260810 04:10:27.876336 31872 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786334794676589-31872-0/minicluster-data/ts-2/wal/instance
Aug 10 04:10:27 dist-test-slave-8w9v krb5kdc[18042](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335027, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.3@KRBTEST.COM for kudu/127.31.32.62@KRBTEST.COM
I20260810 04:10:27.876564 31872 external_mini_cluster.cc:1428] Setting key f559ed58ba5122584025b15c6723ad1c
I20260810 04:10:27.880254 18136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:27.870829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.3:55301 (local address 127.31.32.62:46103)
0810 04:10:27.870900 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0810 04:10:27.870902 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:10:27.871100 (+ 198us) server_negotiation.cc:407] Connection header received
0810 04:10:27.871555 (+ 455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:10:27.871556 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:10:27.871566 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:10:27.871629 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:10:27.872619 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:27.873135 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:27.874126 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:27.874509 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:27.874573 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:10:27.876988 (+ 2415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:10:27.876989 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:10:27.876990 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:10:27.876998 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:10:27.878798 (+ 1800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:10:27.879211 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:10:27.879212 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:10:27.879212 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:10:27.879283 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:10:27.879592 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:10:27.879593 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:10:27.879593 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:10:27.879717 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:10:27.879791 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0810 04:10:27.880045 (+ 254us) server_negotiation.cc:299] Negotiation successful
0810 04:10:27.880112 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260810 04:10:27.880424 18611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:27.870752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:46103 (local address 127.31.32.3:55301)
0810 04:10:27.870939 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0810 04:10:27.870982 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 04:10:27.871468 (+ 486us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:10:27.871679 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:10:27.871683 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:10:27.872340 (+ 657us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:10:27.872534 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:10:27.872539 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:10:27.873215 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:10:27.873215 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:10:27.874013 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:10:27.874015 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:10:27.874612 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:10:27.874613 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:10:27.874729 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:10:27.875207 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:10:27.875216 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:10:27.876881 (+ 1665us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:10:27.878893 (+ 2012us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:10:27.878899 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:10:27.878901 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:10:27.879129 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:10:27.879377 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:10:27.879378 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:10:27.879378 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:10:27.879492 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:10:27.879795 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:10:27.879801 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:10:27.879925 (+ 124us) client_negotiation.cc:769] Sending connection context
0810 04:10:27.880015 (+ 90us) client_negotiation.cc:240] Negotiation successful
0810 04:10:27.880208 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":89,"threads_started":1}
I20260810 04:10:27.880954 18609 heartbeater.cc:347] Connected to a master server at 127.31.32.62:46103
I20260810 04:10:27.881069 18609 heartbeater.cc:464] Registering TS with master...
I20260810 04:10:27.881251 18609 heartbeater.cc:511] Master 127.31.32.62:46103 requested a full tablet report, sending...
I20260810 04:10:27.881624 18091 ts_manager.cc:194] Registered new tserver with Master: cc4f95cac7284b85977f81deb3765372 (127.31.32.3:35761)
I20260810 04:10:27.882005 18091 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.32.3@KRBTEST.COM'} at 127.31.32.3:55301
I20260810 04:10:27.888509 31872 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 04:10:27.896245 18136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:27.889942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39008 (local address 127.31.32.62:46103)
0810 04:10:27.890023 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0810 04:10:27.890026 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:10:27.890141 (+ 115us) server_negotiation.cc:407] Connection header received
0810 04:10:27.890222 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:10:27.890223 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:10:27.890233 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:10:27.890289 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:10:27.890864 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:27.891363 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:27.892098 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:27.892341 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:27.892418 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:10:27.893708 (+ 1290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:10:27.893709 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:10:27.893710 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:10:27.893717 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:10:27.894979 (+ 1262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:10:27.895318 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:10:27.895319 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:10:27.895320 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:10:27.895403 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:10:27.895606 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:10:27.895607 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:10:27.895607 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:10:27.895762 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:10:27.895865 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0810 04:10:27.896030 (+ 165us) server_negotiation.cc:299] Negotiation successful
0810 04:10:27.896106 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260810 04:10:27.897900 18091 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39008:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 04:10:27.899430 18091 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 04:10:27.906225 18623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:27.901893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.3:35761 (local address 127.0.0.1:43546)
0810 04:10:27.902127 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0810 04:10:27.902162 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0810 04:10:27.902226 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:10:27.902929 (+ 703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:10:27.902930 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:10:27.902932 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:10:27.902972 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:10:27.902973 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:10:27.904289 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:10:27.904291 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:10:27.905204 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:10:27.905207 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:10:27.905953 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:10:27.905954 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:10:27.906024 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:10:27.906028 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 04:10:27.906060 (+ 32us) client_negotiation.cc:240] Negotiation successful
0810 04:10:27.906122 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":104,"threads_started":1}
I20260810 04:10:27.906229 18620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:27.902016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.2:38617 (local address 127.0.0.1:47900)
0810 04:10:27.902337 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0810 04:10:27.902347 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 04:10:27.902393 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:10:27.902723 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:10:27.902724 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:10:27.902727 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:10:27.902780 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:10:27.902781 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:10:27.903737 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:10:27.903738 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:10:27.904976 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:10:27.904979 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:10:27.905895 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:10:27.905896 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:10:27.905985 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:10:27.906001 (+ 16us) client_negotiation.cc:769] Sending connection context
0810 04:10:27.906108 (+ 107us) client_negotiation.cc:240] Negotiation successful
0810 04:10:27.906145 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":69,"threads_started":1}
I20260810 04:10:27.906505 18621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:27.901802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.1:44385 (local address 127.0.0.1:40928)
0810 04:10:27.902101 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0810 04:10:27.902146 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 04:10:27.902226 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:10:27.902492 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:10:27.902496 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:10:27.902498 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:10:27.902574 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:10:27.902577 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:10:27.904179 (+ 1602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:10:27.904180 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:10:27.905221 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:10:27.905223 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:10:27.906229 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:10:27.906230 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:10:27.906288 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:10:27.906291 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 04:10:27.906372 (+ 81us) client_negotiation.cc:240] Negotiation successful
0810 04:10:27.906425 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":68,"threads_started":1}
I20260810 04:10:27.906744 18625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:27.902304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47900 (local address 127.31.32.2:38617)
0810 04:10:27.902577 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0810 04:10:27.902581 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 04:10:27.902592 (+ 11us) server_negotiation.cc:407] Connection header received
0810 04:10:27.902629 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:10:27.902631 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:10:27.902647 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:10:27.902716 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:10:27.902851 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:27.903630 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:27.905073 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:27.905791 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:27.905869 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:10:27.905929 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0810 04:10:27.906606 (+ 677us) server_negotiation.cc:299] Negotiation successful
0810 04:10:27.906650 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":233,"thread_start_us":109,"threads_started":1}
I20260810 04:10:27.906755 18624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:27.902009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43546 (local address 127.31.32.3:35761)
0810 04:10:27.902754 (+ 745us) server_negotiation.cc:206] Beginning negotiation
0810 04:10:27.902759 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 04:10:27.902770 (+ 11us) server_negotiation.cc:407] Connection header received
0810 04:10:27.902814 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:10:27.902816 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:10:27.902834 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:10:27.902937 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:10:27.903027 (+ 90us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:27.904093 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:27.905324 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:27.905825 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:27.906473 (+ 648us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:10:27.906547 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0810 04:10:27.906599 (+ 52us) server_negotiation.cc:299] Negotiation successful
0810 04:10:27.906653 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":690,"thread_start_us":102,"threads_started":1}
I20260810 04:10:27.907146 18622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:27.901920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40928 (local address 127.31.32.1:44385)
0810 04:10:27.902298 (+ 378us) server_negotiation.cc:206] Beginning negotiation
0810 04:10:27.902302 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 04:10:27.902313 (+ 11us) server_negotiation.cc:407] Connection header received
0810 04:10:27.902361 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:10:27.902364 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:10:27.902386 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:10:27.902480 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:10:27.902851 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:27.904028 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:27.905312 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:27.906021 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:27.906107 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:10:27.906163 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0810 04:10:27.907002 (+ 839us) server_negotiation.cc:299] Negotiation successful
0810 04:10:27.907062 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":87,"threads_started":1}
I20260810 04:10:27.907591 18535 tablet_service.cc:1467] Processing CreateTablet for tablet 0dcec1aab0ed44bba892c5a3a8813176 (DEFAULT_TABLE table=test-table [id=d43a008ddab34ab4878ee7bb7986f0b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:10:27.907595 18384 tablet_service.cc:1467] Processing CreateTablet for tablet 0dcec1aab0ed44bba892c5a3a8813176 (DEFAULT_TABLE table=test-table [id=d43a008ddab34ab4878ee7bb7986f0b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:10:27.908017 18535 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0dcec1aab0ed44bba892c5a3a8813176. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:10:27.908025 18384 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0dcec1aab0ed44bba892c5a3a8813176. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:10:27.908026 18236 tablet_service.cc:1467] Processing CreateTablet for tablet 0dcec1aab0ed44bba892c5a3a8813176 (DEFAULT_TABLE table=test-table [id=d43a008ddab34ab4878ee7bb7986f0b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:10:27.908418 18236 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0dcec1aab0ed44bba892c5a3a8813176. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:10:27.910940 18626 tablet_bootstrap.cc:492] T 0dcec1aab0ed44bba892c5a3a8813176 P cc4f95cac7284b85977f81deb3765372: Bootstrap starting.
I20260810 04:10:27.911538 18626 tablet_bootstrap.cc:654] T 0dcec1aab0ed44bba892c5a3a8813176 P cc4f95cac7284b85977f81deb3765372: Neither blocks nor log segments found. Creating new log.
I20260810 04:10:27.912010 18627 tablet_bootstrap.cc:492] T 0dcec1aab0ed44bba892c5a3a8813176 P 6e2153c35afe4e3f9eec16136f1220ec: Bootstrap starting.
I20260810 04:10:27.912271 18629 tablet_bootstrap.cc:492] T 0dcec1aab0ed44bba892c5a3a8813176 P 3316ed4624f2494d994309a8b80fe5cc: Bootstrap starting.
I20260810 04:10:27.912370 18626 log.cc:824] T 0dcec1aab0ed44bba892c5a3a8813176 P cc4f95cac7284b85977f81deb3765372: Log is configured to *not* fsync() on all Append() calls
I20260810 04:10:27.912838 18627 tablet_bootstrap.cc:654] T 0dcec1aab0ed44bba892c5a3a8813176 P 6e2153c35afe4e3f9eec16136f1220ec: Neither blocks nor log segments found. Creating new log.
I20260810 04:10:27.912976 18629 tablet_bootstrap.cc:654] T 0dcec1aab0ed44bba892c5a3a8813176 P 3316ed4624f2494d994309a8b80fe5cc: Neither blocks nor log segments found. Creating new log.
I20260810 04:10:27.913431 18626 tablet_bootstrap.cc:492] T 0dcec1aab0ed44bba892c5a3a8813176 P cc4f95cac7284b85977f81deb3765372: No bootstrap required, opened a new log
I20260810 04:10:27.913411 18627 log.cc:824] T 0dcec1aab0ed44bba892c5a3a8813176 P 6e2153c35afe4e3f9eec16136f1220ec: Log is configured to *not* fsync() on all Append() calls
I20260810 04:10:27.913497 18626 ts_tablet_manager.cc:1397] T 0dcec1aab0ed44bba892c5a3a8813176 P cc4f95cac7284b85977f81deb3765372: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260810 04:10:27.913620 18629 log.cc:824] T 0dcec1aab0ed44bba892c5a3a8813176 P 3316ed4624f2494d994309a8b80fe5cc: Log is configured to *not* fsync() on all Append() calls
I20260810 04:10:27.914211 18627 tablet_bootstrap.cc:492] T 0dcec1aab0ed44bba892c5a3a8813176 P 6e2153c35afe4e3f9eec16136f1220ec: No bootstrap required, opened a new log
I20260810 04:10:27.914340 18627 ts_tablet_manager.cc:1397] T 0dcec1aab0ed44bba892c5a3a8813176 P 6e2153c35afe4e3f9eec16136f1220ec: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 04:10:27.914477 18629 tablet_bootstrap.cc:492] T 0dcec1aab0ed44bba892c5a3a8813176 P 3316ed4624f2494d994309a8b80fe5cc: No bootstrap required, opened a new log
I20260810 04:10:27.914546 18629 ts_tablet_manager.cc:1397] T 0dcec1aab0ed44bba892c5a3a8813176 P 3316ed4624f2494d994309a8b80fe5cc: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 04:10:27.915405 18626 raft_consensus.cc:348] T 0dcec1aab0ed44bba892c5a3a8813176 P cc4f95cac7284b85977f81deb3765372 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3316ed4624f2494d994309a8b80fe5cc" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 44385 } } peers { permanent_uuid: "6e2153c35afe4e3f9eec16136f1220ec" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 38617 } } peers { permanent_uuid: "cc4f95cac7284b85977f81deb3765372" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 35761 } }
I20260810 04:10:27.915570 18626 raft_consensus.cc:374] T 0dcec1aab0ed44bba892c5a3a8813176 P cc4f95cac7284b85977f81deb3765372 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:10:27.915614 18626 raft_consensus.cc:729] T 0dcec1aab0ed44bba892c5a3a8813176 P cc4f95cac7284b85977f81deb3765372 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc4f95cac7284b85977f81deb3765372, State: Initialized, Role: FOLLOWER
I20260810 04:10:27.915745 18626 consensus_queue.cc:260] T 0dcec1aab0ed44bba892c5a3a8813176 P cc4f95cac7284b85977f81deb3765372 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3316ed4624f2494d994309a8b80fe5cc" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 44385 } } peers { permanent_uuid: "6e2153c35afe4e3f9eec16136f1220ec" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 38617 } } peers { permanent_uuid: "cc4f95cac7284b85977f81deb3765372" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 35761 } }
I20260810 04:10:27.916078 18609 heartbeater.cc:503] Master 127.31.32.62:46103 was elected leader, sending a full tablet report...
I20260810 04:10:27.916075 18626 ts_tablet_manager.cc:1428] T 0dcec1aab0ed44bba892c5a3a8813176 P cc4f95cac7284b85977f81deb3765372: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260810 04:10:27.916659 18627 raft_consensus.cc:348] T 0dcec1aab0ed44bba892c5a3a8813176 P 6e2153c35afe4e3f9eec16136f1220ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3316ed4624f2494d994309a8b80fe5cc" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 44385 } } peers { permanent_uuid: "6e2153c35afe4e3f9eec16136f1220ec" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 38617 } } peers { permanent_uuid: "cc4f95cac7284b85977f81deb3765372" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 35761 } }
I20260810 04:10:27.916611 18629 raft_consensus.cc:348] T 0dcec1aab0ed44bba892c5a3a8813176 P 3316ed4624f2494d994309a8b80fe5cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3316ed4624f2494d994309a8b80fe5cc" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 44385 } } peers { permanent_uuid: "6e2153c35afe4e3f9eec16136f1220ec" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 38617 } } peers { permanent_uuid: "cc4f95cac7284b85977f81deb3765372" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 35761 } }
I20260810 04:10:27.916827 18627 raft_consensus.cc:374] T 0dcec1aab0ed44bba892c5a3a8813176 P 6e2153c35afe4e3f9eec16136f1220ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:10:27.916827 18629 raft_consensus.cc:374] T 0dcec1aab0ed44bba892c5a3a8813176 P 3316ed4624f2494d994309a8b80fe5cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:10:27.916875 18629 raft_consensus.cc:729] T 0dcec1aab0ed44bba892c5a3a8813176 P 3316ed4624f2494d994309a8b80fe5cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3316ed4624f2494d994309a8b80fe5cc, State: Initialized, Role: FOLLOWER
I20260810 04:10:27.916875 18627 raft_consensus.cc:729] T 0dcec1aab0ed44bba892c5a3a8813176 P 6e2153c35afe4e3f9eec16136f1220ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e2153c35afe4e3f9eec16136f1220ec, State: Initialized, Role: FOLLOWER
I20260810 04:10:27.917012 18629 consensus_queue.cc:260] T 0dcec1aab0ed44bba892c5a3a8813176 P 3316ed4624f2494d994309a8b80fe5cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3316ed4624f2494d994309a8b80fe5cc" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 44385 } } peers { permanent_uuid: "6e2153c35afe4e3f9eec16136f1220ec" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 38617 } } peers { permanent_uuid: "cc4f95cac7284b85977f81deb3765372" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 35761 } }
I20260810 04:10:27.917012 18627 consensus_queue.cc:260] T 0dcec1aab0ed44bba892c5a3a8813176 P 6e2153c35afe4e3f9eec16136f1220ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3316ed4624f2494d994309a8b80fe5cc" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 44385 } } peers { permanent_uuid: "6e2153c35afe4e3f9eec16136f1220ec" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 38617 } } peers { permanent_uuid: "cc4f95cac7284b85977f81deb3765372" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 35761 } }
I20260810 04:10:27.917313 18629 ts_tablet_manager.cc:1428] T 0dcec1aab0ed44bba892c5a3a8813176 P 3316ed4624f2494d994309a8b80fe5cc: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 04:10:27.917330 18457 heartbeater.cc:503] Master 127.31.32.62:46103 was elected leader, sending a full tablet report...
I20260810 04:10:27.917593 18308 heartbeater.cc:503] Master 127.31.32.62:46103 was elected leader, sending a full tablet report...
I20260810 04:10:27.917851 18627 ts_tablet_manager.cc:1428] T 0dcec1aab0ed44bba892c5a3a8813176 P 6e2153c35afe4e3f9eec16136f1220ec: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
W20260810 04:10:27.952395 18458 tablet.cc:2369] T 0dcec1aab0ed44bba892c5a3a8813176 P 6e2153c35afe4e3f9eec16136f1220ec: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 04:10:28.041574 18310 tablet.cc:2369] T 0dcec1aab0ed44bba892c5a3a8813176 P 3316ed4624f2494d994309a8b80fe5cc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 04:10:28.120642 18610 tablet.cc:2369] T 0dcec1aab0ed44bba892c5a3a8813176 P cc4f95cac7284b85977f81deb3765372: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 04:10:28.153484 18635 raft_consensus.cc:482] T 0dcec1aab0ed44bba892c5a3a8813176 P 3316ed4624f2494d994309a8b80fe5cc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 04:10:28.153623 18635 raft_consensus.cc:504] T 0dcec1aab0ed44bba892c5a3a8813176 P 3316ed4624f2494d994309a8b80fe5cc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3316ed4624f2494d994309a8b80fe5cc" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 44385 } } peers { permanent_uuid: "6e2153c35afe4e3f9eec16136f1220ec" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 38617 } } peers { permanent_uuid: "cc4f95cac7284b85977f81deb3765372" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 35761 } }
I20260810 04:10:28.153988 18635 leader_election.cc:288] T 0dcec1aab0ed44bba892c5a3a8813176 P 3316ed4624f2494d994309a8b80fe5cc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6e2153c35afe4e3f9eec16136f1220ec (127.31.32.2:38617), cc4f95cac7284b85977f81deb3765372 (127.31.32.3:35761)
I20260810 04:10:28.157204 18637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:28.154202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.2:38617 (local address 127.31.32.1:53325)
0810 04:10:28.154375 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0810 04:10:28.154385 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 04:10:28.154479 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:10:28.154642 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:10:28.154643 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:10:28.154645 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:10:28.154756 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:10:28.154758 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:10:28.155385 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:10:28.155386 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:10:28.156273 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:10:28.156275 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:10:28.156918 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:10:28.156920 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:10:28.156996 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:10:28.157005 (+ 9us) client_negotiation.cc:769] Sending connection context
0810 04:10:28.157049 (+ 44us) client_negotiation.cc:240] Negotiation successful
0810 04:10:28.157080 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"mutex_wait_us":32,"thread_start_us":100,"threads_started":1}
I20260810 04:10:28.157218 18625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:28.154279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.1:53325 (local address 127.31.32.2:38617)
0810 04:10:28.154327 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0810 04:10:28.154329 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:10:28.154490 (+ 161us) server_negotiation.cc:407] Connection header received
0810 04:10:28.154532 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:10:28.154533 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:10:28.154566 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:10:28.154624 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:10:28.154840 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:28.155284 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:28.156355 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:28.156837 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:28.156895 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:10:28.156904 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0810 04:10:28.157072 (+ 168us) server_negotiation.cc:299] Negotiation successful
0810 04:10:28.157130 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260810 04:10:28.157248 18624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:28.154278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.1:44873 (local address 127.31.32.3:35761)
0810 04:10:28.154343 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0810 04:10:28.154346 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:10:28.154436 (+ 90us) server_negotiation.cc:407] Connection header received
0810 04:10:28.154496 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:10:28.154497 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:10:28.154511 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:10:28.154567 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:10:28.154749 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:28.155275 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:28.156411 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:28.156854 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:28.156943 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:10:28.156953 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0810 04:10:28.157133 (+ 180us) server_negotiation.cc:299] Negotiation successful
0810 04:10:28.157173 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260810 04:10:28.157231 18636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:28.154176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.3:35761 (local address 127.31.32.1:44873)
0810 04:10:28.154349 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0810 04:10:28.154360 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 04:10:28.154431 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:10:28.154585 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:10:28.154586 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:10:28.154588 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:10:28.154674 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:10:28.154677 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:10:28.155382 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:10:28.155383 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:10:28.156302 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:10:28.156304 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:10:28.156959 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:10:28.156960 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:10:28.157030 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:10:28.157052 (+ 22us) client_negotiation.cc:769] Sending connection context
0810 04:10:28.157110 (+ 58us) client_negotiation.cc:240] Negotiation successful
0810 04:10:28.157147 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":75,"threads_started":1}
I20260810 04:10:28.157896 18404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0dcec1aab0ed44bba892c5a3a8813176" candidate_uuid: "3316ed4624f2494d994309a8b80fe5cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e2153c35afe4e3f9eec16136f1220ec" is_pre_election: true
I20260810 04:10:28.157972 18557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0dcec1aab0ed44bba892c5a3a8813176" candidate_uuid: "3316ed4624f2494d994309a8b80fe5cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc4f95cac7284b85977f81deb3765372" is_pre_election: true
I20260810 04:10:28.158066 18404 raft_consensus.cc:2454] T 0dcec1aab0ed44bba892c5a3a8813176 P 6e2153c35afe4e3f9eec16136f1220ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3316ed4624f2494d994309a8b80fe5cc in term 0.
I20260810 04:10:28.158155 18557 raft_consensus.cc:2454] T 0dcec1aab0ed44bba892c5a3a8813176 P cc4f95cac7284b85977f81deb3765372 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3316ed4624f2494d994309a8b80fe5cc in term 0.
I20260810 04:10:28.158325 18184 leader_election.cc:302] T 0dcec1aab0ed44bba892c5a3a8813176 P 3316ed4624f2494d994309a8b80fe5cc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3316ed4624f2494d994309a8b80fe5cc, 6e2153c35afe4e3f9eec16136f1220ec; no voters:
I20260810 04:10:28.158512 18635 raft_consensus.cc:2781] T 0dcec1aab0ed44bba892c5a3a8813176 P 3316ed4624f2494d994309a8b80fe5cc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 04:10:28.158586 18635 raft_consensus.cc:482] T 0dcec1aab0ed44bba892c5a3a8813176 P 3316ed4624f2494d994309a8b80fe5cc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 04:10:28.158604 18635 raft_consensus.cc:3037] T 0dcec1aab0ed44bba892c5a3a8813176 P 3316ed4624f2494d994309a8b80fe5cc [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:10:28.159373 18635 raft_consensus.cc:504] T 0dcec1aab0ed44bba892c5a3a8813176 P 3316ed4624f2494d994309a8b80fe5cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3316ed4624f2494d994309a8b80fe5cc" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 44385 } } peers { permanent_uuid: "6e2153c35afe4e3f9eec16136f1220ec" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 38617 } } peers { permanent_uuid: "cc4f95cac7284b85977f81deb3765372" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 35761 } }
I20260810 04:10:28.159533 18635 leader_election.cc:288] T 0dcec1aab0ed44bba892c5a3a8813176 P 3316ed4624f2494d994309a8b80fe5cc [CANDIDATE]: Term 1 election: Requested vote from peers 6e2153c35afe4e3f9eec16136f1220ec (127.31.32.2:38617), cc4f95cac7284b85977f81deb3765372 (127.31.32.3:35761)
I20260810 04:10:28.159659 18404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0dcec1aab0ed44bba892c5a3a8813176" candidate_uuid: "3316ed4624f2494d994309a8b80fe5cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e2153c35afe4e3f9eec16136f1220ec"
I20260810 04:10:28.159683 18557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0dcec1aab0ed44bba892c5a3a8813176" candidate_uuid: "3316ed4624f2494d994309a8b80fe5cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc4f95cac7284b85977f81deb3765372"
I20260810 04:10:28.159741 18404 raft_consensus.cc:3037] T 0dcec1aab0ed44bba892c5a3a8813176 P 6e2153c35afe4e3f9eec16136f1220ec [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:10:28.159785 18557 raft_consensus.cc:3037] T 0dcec1aab0ed44bba892c5a3a8813176 P cc4f95cac7284b85977f81deb3765372 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:10:28.160566 18404 raft_consensus.cc:2454] T 0dcec1aab0ed44bba892c5a3a8813176 P 6e2153c35afe4e3f9eec16136f1220ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3316ed4624f2494d994309a8b80fe5cc in term 1.
I20260810 04:10:28.160692 18557 raft_consensus.cc:2454] T 0dcec1aab0ed44bba892c5a3a8813176 P cc4f95cac7284b85977f81deb3765372 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3316ed4624f2494d994309a8b80fe5cc in term 1.
I20260810 04:10:28.160754 18184 leader_election.cc:302] T 0dcec1aab0ed44bba892c5a3a8813176 P 3316ed4624f2494d994309a8b80fe5cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3316ed4624f2494d994309a8b80fe5cc, 6e2153c35afe4e3f9eec16136f1220ec; no voters:
I20260810 04:10:28.160857 18635 raft_consensus.cc:2781] T 0dcec1aab0ed44bba892c5a3a8813176 P 3316ed4624f2494d994309a8b80fe5cc [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:10:28.161063 18635 raft_consensus.cc:686] T 0dcec1aab0ed44bba892c5a3a8813176 P 3316ed4624f2494d994309a8b80fe5cc [term 1 LEADER]: Becoming Leader. State: Replica: 3316ed4624f2494d994309a8b80fe5cc, State: Running, Role: LEADER
I20260810 04:10:28.161196 18635 consensus_queue.cc:237] T 0dcec1aab0ed44bba892c5a3a8813176 P 3316ed4624f2494d994309a8b80fe5cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3316ed4624f2494d994309a8b80fe5cc" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 44385 } } peers { permanent_uuid: "6e2153c35afe4e3f9eec16136f1220ec" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 38617 } } peers { permanent_uuid: "cc4f95cac7284b85977f81deb3765372" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 35761 } }
I20260810 04:10:28.162091 18091 catalog_manager.cc:5495] T 0dcec1aab0ed44bba892c5a3a8813176 P 3316ed4624f2494d994309a8b80fe5cc reported cstate change: term changed from 0 to 1, leader changed from <none> to 3316ed4624f2494d994309a8b80fe5cc (127.31.32.1). New cstate: current_term: 1 leader_uuid: "3316ed4624f2494d994309a8b80fe5cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3316ed4624f2494d994309a8b80fe5cc" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 44385 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6e2153c35afe4e3f9eec16136f1220ec" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 38617 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cc4f95cac7284b85977f81deb3765372" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 35761 } health_report { overall_health: UNKNOWN } } }
I20260810 04:10:28.189118 18622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:28.186251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40932 (local address 127.31.32.1:44385)
0810 04:10:28.186322 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0810 04:10:28.186324 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:10:28.186355 (+ 31us) server_negotiation.cc:407] Connection header received
0810 04:10:28.186481 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:10:28.186482 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:10:28.186508 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:10:28.186605 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 04:10:28.186760 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:28.187271 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:28.188085 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:28.188451 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:28.188545 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:10:28.188750 (+ 205us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 04:10:28.188872 (+ 122us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 04:10:28.188977 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0810 04:10:28.188999 (+ 22us) server_negotiation.cc:299] Negotiation successful
0810 04:10:28.189022 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260810 04:10:28.191151 18557 raft_consensus.cc:1261] T 0dcec1aab0ed44bba892c5a3a8813176 P cc4f95cac7284b85977f81deb3765372 [term 1 FOLLOWER]: Refusing update from remote peer 3316ed4624f2494d994309a8b80fe5cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:10:28.191175 18404 raft_consensus.cc:1261] T 0dcec1aab0ed44bba892c5a3a8813176 P 6e2153c35afe4e3f9eec16136f1220ec [term 1 FOLLOWER]: Refusing update from remote peer 3316ed4624f2494d994309a8b80fe5cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:10:28.191623 18638 consensus_queue.cc:1035] T 0dcec1aab0ed44bba892c5a3a8813176 P 3316ed4624f2494d994309a8b80fe5cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc4f95cac7284b85977f81deb3765372" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:10:28.191793 18635 consensus_queue.cc:1035] T 0dcec1aab0ed44bba892c5a3a8813176 P 3316ed4624f2494d994309a8b80fe5cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e2153c35afe4e3f9eec16136f1220ec" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 38617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:10:28.195817 18643 mvcc.cc:205] Tried to move back new op lower bound from 7316828275467751424 to 7316828275348967424. Current Snapshot: MvccSnapshot[applied={T|T < 7316828275467751424 or (T in {7316828275467751424})}]
I20260810 04:10:28.197085 18091 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39008:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 04:10:28.197192 18091 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39008:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 04:10:28.198030 18091 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9830aca959a14e07baa7f8368a283bbb: Sending DeleteTablet for 3 replicas of tablet 0dcec1aab0ed44bba892c5a3a8813176
I20260810 04:10:28.198385 18236 tablet_service.cc:1514] Processing DeleteTablet for tablet 0dcec1aab0ed44bba892c5a3a8813176 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:10:28 UTC) from {username='kudu', principal='kudu/127.31.32.62@KRBTEST.COM'} at 127.0.0.1:40928
I20260810 04:10:28.198479 18384 tablet_service.cc:1514] Processing DeleteTablet for tablet 0dcec1aab0ed44bba892c5a3a8813176 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:10:28 UTC) from {username='kudu', principal='kudu/127.31.32.62@KRBTEST.COM'} at 127.0.0.1:47900
I20260810 04:10:28.198516 18535 tablet_service.cc:1514] Processing DeleteTablet for tablet 0dcec1aab0ed44bba892c5a3a8813176 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:10:28 UTC) from {username='kudu', principal='kudu/127.31.32.62@KRBTEST.COM'} at 127.0.0.1:43546
I20260810 04:10:28.198722 18652 tablet_replica.cc:330] T 0dcec1aab0ed44bba892c5a3a8813176 P 6e2153c35afe4e3f9eec16136f1220ec: stopping tablet replica
I20260810 04:10:28.198837 18653 tablet_replica.cc:330] T 0dcec1aab0ed44bba892c5a3a8813176 P cc4f95cac7284b85977f81deb3765372: stopping tablet replica
I20260810 04:10:28.198879 18652 raft_consensus.cc:2229] T 0dcec1aab0ed44bba892c5a3a8813176 P 6e2153c35afe4e3f9eec16136f1220ec [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 04:10:28.198952 18653 raft_consensus.cc:2229] T 0dcec1aab0ed44bba892c5a3a8813176 P cc4f95cac7284b85977f81deb3765372 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 04:10:28.199067 18653 raft_consensus.cc:2258] T 0dcec1aab0ed44bba892c5a3a8813176 P cc4f95cac7284b85977f81deb3765372 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:10:28.199062 31872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskws89Io/build/release/bin/kudu with pid 18162
I20260810 04:10:28.199244 18652 raft_consensus.cc:2258] T 0dcec1aab0ed44bba892c5a3a8813176 P 6e2153c35afe4e3f9eec16136f1220ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:10:28.199786 18652 ts_tablet_manager.cc:1905] T 0dcec1aab0ed44bba892c5a3a8813176 P 6e2153c35afe4e3f9eec16136f1220ec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:10:28.200901 18653 ts_tablet_manager.cc:1905] T 0dcec1aab0ed44bba892c5a3a8813176 P cc4f95cac7284b85977f81deb3765372: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:10:28.201664 18652 ts_tablet_manager.cc:1918] T 0dcec1aab0ed44bba892c5a3a8813176 P 6e2153c35afe4e3f9eec16136f1220ec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 04:10:28.201735 18652 log.cc:1196] T 0dcec1aab0ed44bba892c5a3a8813176 P 6e2153c35afe4e3f9eec16136f1220ec: Deleting WAL directory at /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786334794676589-31872-0/minicluster-data/ts-1/wal/wals/0dcec1aab0ed44bba892c5a3a8813176
I20260810 04:10:28.201961 18652 ts_tablet_manager.cc:1939] T 0dcec1aab0ed44bba892c5a3a8813176 P 6e2153c35afe4e3f9eec16136f1220ec: Deleting consensus metadata
I20260810 04:10:28.202592 18075 catalog_manager.cc:4841] TS 6e2153c35afe4e3f9eec16136f1220ec (127.31.32.2:38617): tablet 0dcec1aab0ed44bba892c5a3a8813176 (table test-table [id=d43a008ddab34ab4878ee7bb7986f0b4]) successfully deleted
I20260810 04:10:28.203070 18653 ts_tablet_manager.cc:1918] T 0dcec1aab0ed44bba892c5a3a8813176 P cc4f95cac7284b85977f81deb3765372: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 04:10:28.203157 18653 log.cc:1196] T 0dcec1aab0ed44bba892c5a3a8813176 P cc4f95cac7284b85977f81deb3765372: Deleting WAL directory at /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786334794676589-31872-0/minicluster-data/ts-2/wal/wals/0dcec1aab0ed44bba892c5a3a8813176
I20260810 04:10:28.203470 18653 ts_tablet_manager.cc:1939] T 0dcec1aab0ed44bba892c5a3a8813176 P cc4f95cac7284b85977f81deb3765372: Deleting consensus metadata
I20260810 04:10:28.203923 18074 catalog_manager.cc:4841] TS cc4f95cac7284b85977f81deb3765372 (127.31.32.3:35761): tablet 0dcec1aab0ed44bba892c5a3a8813176 (table test-table [id=d43a008ddab34ab4878ee7bb7986f0b4]) successfully deleted
W20260810 04:10:28.207595 18075 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.32.1:44385 (error 108)
I20260810 04:10:28.207926 31872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskws89Io/build/release/bin/kudu with pid 18315
I20260810 04:10:28.208154 18621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:28.208001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.1:44385 (local address 127.0.0.1:40944)
0810 04:10:28.208063 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0810 04:10:28.208098 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.1:44385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260810 04:10:28.210359 18075 catalog_manager.cc:4584] TS 3316ed4624f2494d994309a8b80fe5cc (127.31.32.1:44385): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0dcec1aab0ed44bba892c5a3a8813176: Network error: Client connection negotiation failed: client connection to 127.31.32.1:44385: connect: Connection refused (error 111)
I20260810 04:10:28.215193 31872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskws89Io/build/release/bin/kudu with pid 18462
I20260810 04:10:28.223851 31872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskws89Io/build/release/bin/kudu with pid 18058
2026-08-10T04:10:28Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2219 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-08-10T04:10:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T04:10:28Z Disabled control of system clock
I20260810 04:10:28.253980 31872 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskws89Io/build/release/bin/kudu
/tmp/dist-test-taskws89Io/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786334794676589-31872-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786334794676589-31872-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786334794676589-31872-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786334794676589-31872-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.32.62:35141
--webserver_interface=127.31.32.62
--webserver_port=0
--builtin_ntp_servers=127.31.32.20:44447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.32.62:35141
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 04:10:28.340267 18660 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:10:28.340489 18660 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:10:28.340513 18660 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:10:28.341638 18660 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:10:28.341679 18660 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:10:28.341693 18660 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:10:28.341708 18660 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:10:28.341722 18660 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:10:28.343016 18660 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.32.20:44447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786334794676589-31872-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786334794676589-31872-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.32.62:35141
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.32.62:35141
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786334794676589-31872-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.32.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18660
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786334794676589-31872-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:10:28.343219 18660 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:10:28.343485 18660 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:10:28.348301 18667 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:10:28.348397 18660 server_base.cc:1034] running on GCE node
W20260810 04:10:28.348361 18666 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:10:28.348297 18669 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:10:28.348810 18660 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:10:28.349050 18660 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:10:28.350198 18660 hybrid_clock.cc:648] HybridClock initialized: now 1786335028350191 us; error 30 us; skew 500 ppm
I20260810 04:10:28.355916 18660 webserver.cc:460] Webserver started at http://127.31.32.62:45557/ using document root <none> and password file <none>
I20260810 04:10:28.356259 18660 fs_manager.cc:359] Metadata directory not provided
I20260810 04:10:28.356315 18660 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:10:28.356498 18660 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:10:28.357563 18660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786334794676589-31872-0/minicluster-data/master-0/data/instance:
uuid: "c5dffc7b12dd4ec582ede9ceac7385bb"
format_stamp: "Formatted at 2026-08-10 04:10:28 on dist-test-slave-8w9v"
server_key: "843873c5e693c67c5a154463123314e6"
server_key_iv: "865e52b34c9eac43a5218acf5ffadc86"
server_key_version: "encryptionkey@0"
I20260810 04:10:28.357941 18660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786334794676589-31872-0/minicluster-data/master-0/wal/instance:
uuid: "c5dffc7b12dd4ec582ede9ceac7385bb"
format_stamp: "Formatted at 2026-08-10 04:10:28 on dist-test-slave-8w9v"
server_key: "843873c5e693c67c5a154463123314e6"
server_key_iv: "865e52b34c9eac43a5218acf5ffadc86"
server_key_version: "encryptionkey@0"
I20260810 04:10:28.359277 18660 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 04:10:28.360008 18675 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:10:28.360219 18660 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 04:10:28.360320 18660 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786334794676589-31872-0/minicluster-data/master-0/data,/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786334794676589-31872-0/minicluster-data/master-0/wal
uuid: "c5dffc7b12dd4ec582ede9ceac7385bb"
format_stamp: "Formatted at 2026-08-10 04:10:28 on dist-test-slave-8w9v"
server_key: "843873c5e693c67c5a154463123314e6"
server_key_iv: "865e52b34c9eac43a5218acf5ffadc86"
server_key_version: "encryptionkey@0"
I20260810 04:10:28.360426 18660 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786334794676589-31872-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786334794676589-31872-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786334794676589-31872-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:10:28.387395 18660 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:10:28.387776 18660 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:10:28.387945 18660 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:10:28.392313 18660 rpc_server.cc:307] RPC server started. Bound to: 127.31.32.62:35141
I20260810 04:10:28.392370 18734 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.32.62:35141 every 8 connection(s)
I20260810 04:10:28.392758 18660 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786334794676589-31872-0/minicluster-data/master-0/data/info.pb
I20260810 04:10:28.393882 18735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:10:28.396474 18735 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5dffc7b12dd4ec582ede9ceac7385bb: Bootstrap starting.
I20260810 04:10:28.397079 18735 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c5dffc7b12dd4ec582ede9ceac7385bb: Neither blocks nor log segments found. Creating new log.
I20260810 04:10:28.397408 18735 log.cc:824] T 00000000000000000000000000000000 P c5dffc7b12dd4ec582ede9ceac7385bb: Log is configured to *not* fsync() on all Append() calls
I20260810 04:10:28.398131 18735 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5dffc7b12dd4ec582ede9ceac7385bb: No bootstrap required, opened a new log
I20260810 04:10:28.398766 31872 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskws89Io/build/release/bin/kudu as pid 18660
I20260810 04:10:28.398906 31872 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786334794676589-31872-0/minicluster-data/master-0/wal/instance
I20260810 04:10:28.399113 31872 external_mini_cluster.cc:1428] Setting key ae1259efccb9ec56703f6e4938193ecc
I20260810 04:10:28.400468 18735 raft_consensus.cc:348] T 00000000000000000000000000000000 P c5dffc7b12dd4ec582ede9ceac7385bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5dffc7b12dd4ec582ede9ceac7385bb" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 35141 } }
I20260810 04:10:28.400651 18735 raft_consensus.cc:374] T 00000000000000000000000000000000 P c5dffc7b12dd4ec582ede9ceac7385bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:10:28.400689 18735 raft_consensus.cc:729] T 00000000000000000000000000000000 P c5dffc7b12dd4ec582ede9ceac7385bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5dffc7b12dd4ec582ede9ceac7385bb, State: Initialized, Role: FOLLOWER
I20260810 04:10:28.400821 18735 consensus_queue.cc:260] T 00000000000000000000000000000000 P c5dffc7b12dd4ec582ede9ceac7385bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5dffc7b12dd4ec582ede9ceac7385bb" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 35141 } }
I20260810 04:10:28.400928 18735 raft_consensus.cc:388] T 00000000000000000000000000000000 P c5dffc7b12dd4ec582ede9ceac7385bb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 04:10:28.400971 18735 raft_consensus.cc:482] T 00000000000000000000000000000000 P c5dffc7b12dd4ec582ede9ceac7385bb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 04:10:28.401011 18735 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c5dffc7b12dd4ec582ede9ceac7385bb [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:10:28.401827 18735 raft_consensus.cc:504] T 00000000000000000000000000000000 P c5dffc7b12dd4ec582ede9ceac7385bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5dffc7b12dd4ec582ede9ceac7385bb" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 35141 } }
I20260810 04:10:28.401948 18735 leader_election.cc:302] T 00000000000000000000000000000000 P c5dffc7b12dd4ec582ede9ceac7385bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c5dffc7b12dd4ec582ede9ceac7385bb; no voters:
I20260810 04:10:28.402129 18735 leader_election.cc:288] T 00000000000000000000000000000000 P c5dffc7b12dd4ec582ede9ceac7385bb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 04:10:28.402384 18735 sys_catalog.cc:564] T 00000000000000000000000000000000 P c5dffc7b12dd4ec582ede9ceac7385bb [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:10:28.402994 18741 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c5dffc7b12dd4ec582ede9ceac7385bb [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:10:28.403136 18741 raft_consensus.cc:686] T 00000000000000000000000000000000 P c5dffc7b12dd4ec582ede9ceac7385bb [term 1 LEADER]: Becoming Leader. State: Replica: c5dffc7b12dd4ec582ede9ceac7385bb, State: Running, Role: LEADER
I20260810 04:10:28.403239 18741 consensus_queue.cc:237] T 00000000000000000000000000000000 P c5dffc7b12dd4ec582ede9ceac7385bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5dffc7b12dd4ec582ede9ceac7385bb" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 35141 } }
I20260810 04:10:28.403601 18741 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5dffc7b12dd4ec582ede9ceac7385bb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c5dffc7b12dd4ec582ede9ceac7385bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5dffc7b12dd4ec582ede9ceac7385bb" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 35141 } } }
I20260810 04:10:28.403695 18741 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5dffc7b12dd4ec582ede9ceac7385bb [sys.catalog]: This master's current role is: LEADER
I20260810 04:10:28.403820 18741 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5dffc7b12dd4ec582ede9ceac7385bb [sys.catalog]: SysCatalogTable state changed. Reason: New leader c5dffc7b12dd4ec582ede9ceac7385bb. Latest consensus state: current_term: 1 leader_uuid: "c5dffc7b12dd4ec582ede9ceac7385bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5dffc7b12dd4ec582ede9ceac7385bb" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 35141 } } }
I20260810 04:10:28.403940 18757 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 04:10:28.404031 18741 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5dffc7b12dd4ec582ede9ceac7385bb [sys.catalog]: This master's current role is: LEADER
I20260810 04:10:28.404525 18757 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 04:10:28.406219 18757 catalog_manager.cc:1347] Generated new cluster ID: 3ce891be8269407d9945a2f55ebb9d85
I20260810 04:10:28.406296 18757 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 04:10:28.406762 18739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:28.399686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54348 (local address 127.31.32.62:35141)
0810 04:10:28.399933 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0810 04:10:28.399943 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 04:10:28.399982 (+ 39us) server_negotiation.cc:407] Connection header received
0810 04:10:28.400422 (+ 440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:10:28.400431 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:10:28.400520 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:10:28.400681 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:10:28.400970 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:28.402814 (+ 1844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:28.404241 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:28.404830 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:28.404905 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:10:28.405789 (+ 884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:10:28.405793 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:10:28.405793 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:10:28.406012 (+ 219us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:10:28.406054 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:10:28.406069 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:10:28.406416 (+ 347us) server_negotiation.cc:1035] Waiting for connection context
0810 04:10:28.406461 (+ 45us) server_negotiation.cc:299] Negotiation successful
0810 04:10:28.406544 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":82,"threads_started":1}
I20260810 04:10:28.414489 18757 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 04:10:28.415246 18757 catalog_manager.cc:1504] Loading token signing keys...
I20260810 04:10:28.426080 18757 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c5dffc7b12dd4ec582ede9ceac7385bb: Generated new TSK 0
I20260810 04:10:28.426297 18757 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 04:10:28.433538 31872 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskws89Io/build/release/bin/kudu
/tmp/dist-test-taskws89Io/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786334794676589-31872-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786334794676589-31872-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786334794676589-31872-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786334794676589-31872-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.32.1:0
--local_ip_for_outbound_sockets=127.31.32.1
--webserver_interface=127.31.32.1
--webserver_port=0
--tserver_master_addrs=127.31.32.62:35141
--builtin_ntp_servers=127.31.32.20:44447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 04:10:28.519783 18763 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:10:28.520031 18763 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:10:28.520071 18763 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:10:28.521291 18763 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:10:28.521366 18763 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:10:28.521409 18763 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.32.1
I20260810 04:10:28.522827 18763 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.32.20:44447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786334794676589-31872-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786334794676589-31872-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.32.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786334794676589-31872-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.32.1
--webserver_port=0
--tserver_master_addrs=127.31.32.62:35141
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18763
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.32.1
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786334794676589-31872-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:10:28.523108 18763 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:10:28.523442 18763 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:10:28.529580 18769 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:10:28.529585 18771 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:10:28.529829 18763 server_base.cc:1034] running on GCE node
W20260810 04:10:28.529613 18768 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:10:28.530110 18763 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:10:28.530386 18763 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:10:28.531566 18763 hybrid_clock.cc:648] HybridClock initialized: now 1786335028531549 us; error 33 us; skew 500 ppm
I20260810 04:10:28.532874 18763 webserver.cc:460] Webserver started at http://127.31.32.1:44697/ using document root <none> and password file <none>
I20260810 04:10:28.533131 18763 fs_manager.cc:359] Metadata directory not provided
I20260810 04:10:28.533203 18763 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:10:28.533357 18763 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:10:28.534446 18763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786334794676589-31872-0/minicluster-data/ts-0/data/instance:
uuid: "0b3bbbe7b27f410cab32a1123b24a7e2"
format_stamp: "Formatted at 2026-08-10 04:10:28 on dist-test-slave-8w9v"
server_key: "6b981b79939767bac3d5848eccd7b0cc"
server_key_iv: "3946db4b09e6d13350a3b850ec4813d3"
server_key_version: "encryptionkey@0"
I20260810 04:10:28.534806 18763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786334794676589-31872-0/minicluster-data/ts-0/wal/instance:
uuid: "0b3bbbe7b27f410cab32a1123b24a7e2"
format_stamp: "Formatted at 2026-08-10 04:10:28 on dist-test-slave-8w9v"
server_key: "6b981b79939767bac3d5848eccd7b0cc"
server_key_iv: "3946db4b09e6d13350a3b850ec4813d3"
server_key_version: "encryptionkey@0"
I20260810 04:10:28.536199 18763 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 04:10:28.536962 18777 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:10:28.537142 18763 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 04:10:28.537249 18763 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786334794676589-31872-0/minicluster-data/ts-0/data,/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786334794676589-31872-0/minicluster-data/ts-0/wal
uuid: "0b3bbbe7b27f410cab32a1123b24a7e2"
format_stamp: "Formatted at 2026-08-10 04:10:28 on dist-test-slave-8w9v"
server_key: "6b981b79939767bac3d5848eccd7b0cc"
server_key_iv: "3946db4b09e6d13350a3b850ec4813d3"
server_key_version: "encryptionkey@0"
I20260810 04:10:28.537330 18763 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786334794676589-31872-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786334794676589-31872-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786334794676589-31872-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:10:28.558383 18763 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:10:28.558727 18763 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:10:28.558890 18763 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:10:28.559139 18763 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:10:28.559492 18763 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:10:28.559556 18763 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:10:28.559600 18763 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:10:28.559625 18763 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:10:28.565691 18763 rpc_server.cc:307] RPC server started. Bound to: 127.31.32.1:32937
I20260810 04:10:28.565798 18905 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.32.1:32937 every 8 connection(s)
I20260810 04:10:28.566126 18763 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786334794676589-31872-0/minicluster-data/ts-0/data/info.pb
I20260810 04:10:28.569391 31872 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskws89Io/build/release/bin/kudu as pid 18763
I20260810 04:10:28.569525 31872 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786334794676589-31872-0/minicluster-data/ts-0/wal/instance
I20260810 04:10:28.569722 31872 external_mini_cluster.cc:1428] Setting key 41b23153b9bd4d90e9ffaea4e6fd9ae6
I20260810 04:10:28.570737 31872 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskws89Io/build/release/bin/kudu
/tmp/dist-test-taskws89Io/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786334794676589-31872-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786334794676589-31872-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786334794676589-31872-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786334794676589-31872-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.32.2:0
--local_ip_for_outbound_sockets=127.31.32.2
--webserver_interface=127.31.32.2
--webserver_port=0
--tserver_master_addrs=127.31.32.62:35141
--builtin_ntp_servers=127.31.32.20:44447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 04:10:28.572244 18739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:28.566931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.1:52547 (local address 127.31.32.62:35141)
0810 04:10:28.567021 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0810 04:10:28.567023 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:10:28.567223 (+ 200us) server_negotiation.cc:407] Connection header received
0810 04:10:28.567725 (+ 502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:10:28.567726 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:10:28.567739 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:10:28.567805 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:10:28.568572 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:28.569092 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:28.570629 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:28.570907 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:28.570986 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:10:28.571901 (+ 915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:10:28.571902 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:10:28.571902 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:10:28.571907 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:10:28.571921 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:10:28.571924 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:10:28.571969 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0810 04:10:28.572105 (+ 136us) server_negotiation.cc:299] Negotiation successful
0810 04:10:28.572143 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 04:10:28.572521 18908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:28.566890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:35141 (local address 127.31.32.1:52547)
0810 04:10:28.567082 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0810 04:10:28.567121 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0810 04:10:28.567610 (+ 489us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:10:28.567863 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:10:28.567867 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:10:28.567875 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:10:28.568466 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:10:28.568471 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:10:28.569239 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:10:28.569241 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:10:28.570544 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:10:28.570547 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:10:28.570994 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:10:28.570995 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:10:28.571157 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:10:28.571735 (+ 578us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 04:10:28.571736 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:10:28.571775 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 04:10:28.571804 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:10:28.572018 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:10:28.572025 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:10:28.572028 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 04:10:28.572106 (+ 78us) client_negotiation.cc:240] Negotiation successful
0810 04:10:28.572240 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":106,"threads_started":1}
I20260810 04:10:28.572981 18906 heartbeater.cc:347] Connected to a master server at 127.31.32.62:35141
I20260810 04:10:28.573072 18906 heartbeater.cc:464] Registering TS with master...
I20260810 04:10:28.573280 18906 heartbeater.cc:511] Master 127.31.32.62:35141 requested a full tablet report, sending...
I20260810 04:10:28.573786 18695 ts_manager.cc:194] Registered new tserver with Master: 0b3bbbe7b27f410cab32a1123b24a7e2 (127.31.32.1:32937)
I20260810 04:10:28.574416 18695 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.32.1:52547
W20260810 04:10:28.656066 18909 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:10:28.656347 18909 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:10:28.656384 18909 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:10:28.657544 18909 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:10:28.657614 18909 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:10:28.657657 18909 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.32.2
I20260810 04:10:28.659080 18909 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.32.20:44447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786334794676589-31872-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786334794676589-31872-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.32.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786334794676589-31872-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.32.2
--webserver_port=0
--tserver_master_addrs=127.31.32.62:35141
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18909
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.32.2
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786334794676589-31872-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:10:28.659359 18909 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:10:28.659694 18909 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:10:28.665040 18916 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:10:28.665031 18919 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:10:28.665060 18909 server_base.cc:1034] running on GCE node
W20260810 04:10:28.665031 18915 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:10:28.665421 18909 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:10:28.665694 18909 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:10:28.666843 18909 hybrid_clock.cc:648] HybridClock initialized: now 1786335028666827 us; error 43 us; skew 500 ppm
I20260810 04:10:28.668303 18909 webserver.cc:460] Webserver started at http://127.31.32.2:42377/ using document root <none> and password file <none>
I20260810 04:10:28.668593 18909 fs_manager.cc:359] Metadata directory not provided
I20260810 04:10:28.668643 18909 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:10:28.668783 18909 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:10:28.669916 18909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786334794676589-31872-0/minicluster-data/ts-1/data/instance:
uuid: "7f10750c69d14b6d85496fee617cf5d0"
format_stamp: "Formatted at 2026-08-10 04:10:28 on dist-test-slave-8w9v"
server_key: "a9dc304c8e2eea5db381b50820b1d922"
server_key_iv: "fd41ec9834639ff5ab10f76dc8e1947f"
server_key_version: "encryptionkey@0"
I20260810 04:10:28.670295 18909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786334794676589-31872-0/minicluster-data/ts-1/wal/instance:
uuid: "7f10750c69d14b6d85496fee617cf5d0"
format_stamp: "Formatted at 2026-08-10 04:10:28 on dist-test-slave-8w9v"
server_key: "a9dc304c8e2eea5db381b50820b1d922"
server_key_iv: "fd41ec9834639ff5ab10f76dc8e1947f"
server_key_version: "encryptionkey@0"
I20260810 04:10:28.671679 18909 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 04:10:28.672516 18925 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:10:28.672696 18909 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260810 04:10:28.672806 18909 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786334794676589-31872-0/minicluster-data/ts-1/data,/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786334794676589-31872-0/minicluster-data/ts-1/wal
uuid: "7f10750c69d14b6d85496fee617cf5d0"
format_stamp: "Formatted at 2026-08-10 04:10:28 on dist-test-slave-8w9v"
server_key: "a9dc304c8e2eea5db381b50820b1d922"
server_key_iv: "fd41ec9834639ff5ab10f76dc8e1947f"
server_key_version: "encryptionkey@0"
I20260810 04:10:28.672894 18909 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786334794676589-31872-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786334794676589-31872-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786334794676589-31872-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:10:28.687881 18909 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:10:28.688323 18909 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:10:28.688489 18909 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:10:28.688730 18909 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:10:28.689111 18909 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:10:28.689148 18909 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:10:28.689172 18909 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:10:28.689221 18909 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:10:28.695026 18909 rpc_server.cc:307] RPC server started. Bound to: 127.31.32.2:40239
I20260810 04:10:28.695098 19049 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.32.2:40239 every 8 connection(s)
I20260810 04:10:28.695468 18909 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786334794676589-31872-0/minicluster-data/ts-1/data/info.pb
I20260810 04:10:28.696882 31872 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskws89Io/build/release/bin/kudu as pid 18909
I20260810 04:10:28.696991 31872 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786334794676589-31872-0/minicluster-data/ts-1/wal/instance
I20260810 04:10:28.697227 31872 external_mini_cluster.cc:1428] Setting key 83f61a66a404c07799ab9f220a9bf308
I20260810 04:10:28.698277 31872 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskws89Io/build/release/bin/kudu
/tmp/dist-test-taskws89Io/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786334794676589-31872-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786334794676589-31872-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786334794676589-31872-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786334794676589-31872-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.32.3:0
--local_ip_for_outbound_sockets=127.31.32.3
--webserver_interface=127.31.32.3
--webserver_port=0
--tserver_master_addrs=127.31.32.62:35141
--builtin_ntp_servers=127.31.32.20:44447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 04:10:28.701460 18739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:28.696390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.2:51801 (local address 127.31.32.62:35141)
0810 04:10:28.696439 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0810 04:10:28.696442 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:10:28.696695 (+ 253us) server_negotiation.cc:407] Connection header received
0810 04:10:28.697391 (+ 696us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:10:28.697392 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:10:28.697407 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:10:28.697491 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:10:28.698362 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:28.698879 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:28.699869 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:28.700087 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:28.700204 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:10:28.701118 (+ 914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:10:28.701120 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:10:28.701120 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:10:28.701126 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:10:28.701142 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:10:28.701146 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:10:28.701215 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0810 04:10:28.701349 (+ 134us) server_negotiation.cc:299] Negotiation successful
0810 04:10:28.701379 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260810 04:10:28.701725 19052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:28.696372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:35141 (local address 127.31.32.2:51801)
0810 04:10:28.696561 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0810 04:10:28.696597 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0810 04:10:28.697281 (+ 684us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:10:28.697573 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:10:28.697578 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:10:28.697591 (+ 13us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:10:28.698233 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:10:28.698239 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:10:28.698978 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:10:28.698979 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:10:28.699775 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:10:28.699777 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:10:28.700217 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:10:28.700218 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:10:28.700369 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:10:28.700968 (+ 599us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 04:10:28.700968 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:10:28.700998 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 04:10:28.701022 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:10:28.701217 (+ 195us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:10:28.701223 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:10:28.701226 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 04:10:28.701337 (+ 111us) client_negotiation.cc:240] Negotiation successful
0810 04:10:28.701441 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":81,"threads_started":1}
I20260810 04:10:28.702270 19050 heartbeater.cc:347] Connected to a master server at 127.31.32.62:35141
I20260810 04:10:28.702380 19050 heartbeater.cc:464] Registering TS with master...
I20260810 04:10:28.702567 19050 heartbeater.cc:511] Master 127.31.32.62:35141 requested a full tablet report, sending...
I20260810 04:10:28.703037 18695 ts_manager.cc:194] Registered new tserver with Master: 7f10750c69d14b6d85496fee617cf5d0 (127.31.32.2:40239)
I20260810 04:10:28.703440 18695 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.32.2:51801
W20260810 04:10:28.784368 19053 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:10:28.784572 19053 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:10:28.784595 19053 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:10:28.785748 19053 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:10:28.785789 19053 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:10:28.785844 19053 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.32.3
I20260810 04:10:28.787213 19053 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.32.20:44447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786334794676589-31872-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786334794676589-31872-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.32.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786334794676589-31872-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.32.3
--webserver_port=0
--tserver_master_addrs=127.31.32.62:35141
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19053
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.32.3
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786334794676589-31872-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:10:28.787472 19053 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:10:28.787781 19053 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:10:28.793107 19058 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:10:28.793133 19061 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:10:28.793147 19059 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:10:28.793332 19053 server_base.cc:1034] running on GCE node
I20260810 04:10:28.793553 19053 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:10:28.793805 19053 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:10:28.795011 19053 hybrid_clock.cc:648] HybridClock initialized: now 1786335028794991 us; error 33 us; skew 500 ppm
I20260810 04:10:28.796648 19053 webserver.cc:460] Webserver started at http://127.31.32.3:38073/ using document root <none> and password file <none>
I20260810 04:10:28.796919 19053 fs_manager.cc:359] Metadata directory not provided
I20260810 04:10:28.796998 19053 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:10:28.797178 19053 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:10:28.798345 19053 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786334794676589-31872-0/minicluster-data/ts-2/data/instance:
uuid: "1e1d21a1e555413fbd33ae2b59528073"
format_stamp: "Formatted at 2026-08-10 04:10:28 on dist-test-slave-8w9v"
server_key: "1872fe4741a805747eded19d7c3cc836"
server_key_iv: "5995d45a255891300e8fadde48f94058"
server_key_version: "encryptionkey@0"
I20260810 04:10:28.798769 19053 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786334794676589-31872-0/minicluster-data/ts-2/wal/instance:
uuid: "1e1d21a1e555413fbd33ae2b59528073"
format_stamp: "Formatted at 2026-08-10 04:10:28 on dist-test-slave-8w9v"
server_key: "1872fe4741a805747eded19d7c3cc836"
server_key_iv: "5995d45a255891300e8fadde48f94058"
server_key_version: "encryptionkey@0"
I20260810 04:10:28.800290 19053 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.003s
I20260810 04:10:28.801105 19067 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:10:28.801277 19053 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 04:10:28.801407 19053 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786334794676589-31872-0/minicluster-data/ts-2/data,/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786334794676589-31872-0/minicluster-data/ts-2/wal
uuid: "1e1d21a1e555413fbd33ae2b59528073"
format_stamp: "Formatted at 2026-08-10 04:10:28 on dist-test-slave-8w9v"
server_key: "1872fe4741a805747eded19d7c3cc836"
server_key_iv: "5995d45a255891300e8fadde48f94058"
server_key_version: "encryptionkey@0"
I20260810 04:10:28.801484 19053 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786334794676589-31872-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786334794676589-31872-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786334794676589-31872-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:10:28.813247 19053 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:10:28.813584 19053 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:10:28.813753 19053 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:10:28.814028 19053 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:10:28.814370 19053 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:10:28.814425 19053 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:10:28.814460 19053 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:10:28.814488 19053 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:10:28.820677 19053 rpc_server.cc:307] RPC server started. Bound to: 127.31.32.3:37185
I20260810 04:10:28.820742 19194 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.32.3:37185 every 8 connection(s)
I20260810 04:10:28.821113 19053 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786334794676589-31872-0/minicluster-data/ts-2/data/info.pb
I20260810 04:10:28.824288 31872 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskws89Io/build/release/bin/kudu as pid 19053
I20260810 04:10:28.824399 31872 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786334794676589-31872-0/minicluster-data/ts-2/wal/instance
I20260810 04:10:28.824602 31872 external_mini_cluster.cc:1428] Setting key 3258d46d6b822f5e54f4fbb75616e21c
I20260810 04:10:28.826982 18739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:28.821894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.3:45375 (local address 127.31.32.62:35141)
0810 04:10:28.821963 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0810 04:10:28.821965 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:10:28.822234 (+ 269us) server_negotiation.cc:407] Connection header received
0810 04:10:28.822744 (+ 510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:10:28.822745 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:10:28.822755 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:10:28.822813 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:10:28.823461 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:28.823981 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:28.825536 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:28.825830 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:28.825892 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:10:28.826762 (+ 870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:10:28.826764 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:10:28.826764 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:10:28.826768 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:10:28.826784 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:10:28.826788 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:10:28.826841 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0810 04:10:28.826898 (+ 57us) server_negotiation.cc:299] Negotiation successful
0810 04:10:28.826922 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260810 04:10:28.827247 19197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:28.821887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:35141 (local address 127.31.32.3:45375)
0810 04:10:28.822084 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0810 04:10:28.822119 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0810 04:10:28.822659 (+ 540us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:10:28.822868 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:10:28.822871 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:10:28.822879 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:10:28.823379 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:10:28.823383 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:10:28.824081 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:10:28.824082 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:10:28.825440 (+ 1358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:10:28.825442 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:10:28.825911 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:10:28.825912 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:10:28.826024 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:10:28.826617 (+ 593us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 04:10:28.826618 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:10:28.826649 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 04:10:28.826672 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:10:28.826848 (+ 176us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:10:28.826851 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:10:28.826853 (+ 2us) client_negotiation.cc:769] Sending connection context
0810 04:10:28.826899 (+ 46us) client_negotiation.cc:240] Negotiation successful
0810 04:10:28.826975 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":91,"threads_started":1}
I20260810 04:10:28.827780 19195 heartbeater.cc:347] Connected to a master server at 127.31.32.62:35141
I20260810 04:10:28.827878 19195 heartbeater.cc:464] Registering TS with master...
I20260810 04:10:28.828030 19195 heartbeater.cc:511] Master 127.31.32.62:35141 requested a full tablet report, sending...
I20260810 04:10:28.828473 18695 ts_manager.cc:194] Registered new tserver with Master: 1e1d21a1e555413fbd33ae2b59528073 (127.31.32.3:37185)
I20260810 04:10:28.828869 18695 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.32.3:45375
I20260810 04:10:28.836798 31872 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 04:10:28.841254 18739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:28.838325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54350 (local address 127.31.32.62:35141)
0810 04:10:28.838399 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0810 04:10:28.838402 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:10:28.838504 (+ 102us) server_negotiation.cc:407] Connection header received
0810 04:10:28.838561 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:10:28.838562 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:10:28.838574 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:10:28.838623 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:10:28.838784 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:28.839252 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:28.839993 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:28.840221 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:28.840276 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:10:28.841022 (+ 746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:10:28.841022 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:10:28.841023 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:10:28.841027 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:10:28.841040 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:10:28.841043 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:10:28.841080 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0810 04:10:28.841123 (+ 43us) server_negotiation.cc:299] Negotiation successful
0810 04:10:28.841144 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260810 04:10:28.842825 18695 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54350:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 04:10:28.844470 18695 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 04:10:28.851111 19203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:28.846968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.3:37185 (local address 127.0.0.1:34596)
0810 04:10:28.847273 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0810 04:10:28.847284 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 04:10:28.847346 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:10:28.847660 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:10:28.847663 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:10:28.847666 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:10:28.847729 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:10:28.847731 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:10:28.848843 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:10:28.848844 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:10:28.849984 (+ 1140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:10:28.849987 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:10:28.850787 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:10:28.850788 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:10:28.850876 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:10:28.850892 (+ 16us) client_negotiation.cc:769] Sending connection context
0810 04:10:28.850968 (+ 76us) client_negotiation.cc:240] Negotiation successful
0810 04:10:28.851024 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":74,"threads_started":1}
I20260810 04:10:28.851122 19206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:28.847073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34596 (local address 127.31.32.3:37185)
0810 04:10:28.847430 (+ 357us) server_negotiation.cc:206] Beginning negotiation
0810 04:10:28.847435 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 04:10:28.847446 (+ 11us) server_negotiation.cc:407] Connection header received
0810 04:10:28.847499 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:10:28.847501 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:10:28.847560 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:10:28.847656 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:10:28.847923 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:28.848747 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:28.850061 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:28.850696 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:28.850772 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:10:28.850871 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0810 04:10:28.850998 (+ 127us) server_negotiation.cc:299] Negotiation successful
0810 04:10:28.851036 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":85,"threads_started":1}
I20260810 04:10:28.851893 19204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:28.846965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.1:32937 (local address 127.0.0.1:45142)
0810 04:10:28.847157 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0810 04:10:28.847204 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0810 04:10:28.847269 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:10:28.847981 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:10:28.847982 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:10:28.847983 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:10:28.848014 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:10:28.848015 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:10:28.849307 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:10:28.849308 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:10:28.850257 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:10:28.850259 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:10:28.851647 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:10:28.851648 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:10:28.851696 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:10:28.851699 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 04:10:28.851775 (+ 76us) client_negotiation.cc:240] Negotiation successful
0810 04:10:28.851824 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":133,"threads_started":1}
I20260810 04:10:28.851909 19208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:28.847190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45142 (local address 127.31.32.1:32937)
0810 04:10:28.847809 (+ 619us) server_negotiation.cc:206] Beginning negotiation
0810 04:10:28.847813 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 04:10:28.847824 (+ 11us) server_negotiation.cc:407] Connection header received
0810 04:10:28.847868 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:10:28.847869 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:10:28.847886 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:10:28.847971 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:10:28.848085 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:28.849220 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:28.850344 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:28.851091 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:28.851500 (+ 409us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:10:28.851603 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0810 04:10:28.851792 (+ 189us) server_negotiation.cc:299] Negotiation successful
0810 04:10:28.851832 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":576,"thread_start_us":97,"threads_started":1}
I20260810 04:10:28.851990 19120 tablet_service.cc:1467] Processing CreateTablet for tablet 51027c9d845e4cd48fff13aa352bf1d7 (DEFAULT_TABLE table=test-table [id=326d13f07aed4c7eb6e848a39aa18cea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:10:28.852337 19207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:28.847186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.2:40239 (local address 127.0.0.1:47058)
0810 04:10:28.847419 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0810 04:10:28.847430 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 04:10:28.847484 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:10:28.847797 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:10:28.847798 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:10:28.847800 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:10:28.847839 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:10:28.847841 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:10:28.850356 (+ 2515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:10:28.850356 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:10:28.851176 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:10:28.851179 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:10:28.852086 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:10:28.852087 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:10:28.852173 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:10:28.852177 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 04:10:28.852231 (+ 54us) client_negotiation.cc:240] Negotiation successful
0810 04:10:28.852271 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":75,"threads_started":1}
I20260810 04:10:28.852437 19120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51027c9d845e4cd48fff13aa352bf1d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:10:28.852576 19209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:28.847398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47058 (local address 127.31.32.2:40239)
0810 04:10:28.847623 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0810 04:10:28.847628 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 04:10:28.847639 (+ 11us) server_negotiation.cc:407] Connection header received
0810 04:10:28.847683 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:10:28.847686 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:10:28.847701 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:10:28.847797 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:10:28.847976 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:28.849196 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:28.851254 (+ 2058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:28.851891 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:28.851961 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:10:28.852045 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0810 04:10:28.852447 (+ 402us) server_negotiation.cc:299] Negotiation successful
0810 04:10:28.852494 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":106,"threads_started":1}
I20260810 04:10:28.852968 18832 tablet_service.cc:1467] Processing CreateTablet for tablet 51027c9d845e4cd48fff13aa352bf1d7 (DEFAULT_TABLE table=test-table [id=326d13f07aed4c7eb6e848a39aa18cea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:10:28.853197 18978 tablet_service.cc:1467] Processing CreateTablet for tablet 51027c9d845e4cd48fff13aa352bf1d7 (DEFAULT_TABLE table=test-table [id=326d13f07aed4c7eb6e848a39aa18cea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:10:28.853303 18832 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51027c9d845e4cd48fff13aa352bf1d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:10:28.853497 18978 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51027c9d845e4cd48fff13aa352bf1d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:10:28.855018 19210 tablet_bootstrap.cc:492] T 51027c9d845e4cd48fff13aa352bf1d7 P 1e1d21a1e555413fbd33ae2b59528073: Bootstrap starting.
I20260810 04:10:28.855608 19210 tablet_bootstrap.cc:654] T 51027c9d845e4cd48fff13aa352bf1d7 P 1e1d21a1e555413fbd33ae2b59528073: Neither blocks nor log segments found. Creating new log.
I20260810 04:10:28.855841 19210 log.cc:824] T 51027c9d845e4cd48fff13aa352bf1d7 P 1e1d21a1e555413fbd33ae2b59528073: Log is configured to *not* fsync() on all Append() calls
I20260810 04:10:28.856561 19210 tablet_bootstrap.cc:492] T 51027c9d845e4cd48fff13aa352bf1d7 P 1e1d21a1e555413fbd33ae2b59528073: No bootstrap required, opened a new log
I20260810 04:10:28.856626 19210 ts_tablet_manager.cc:1397] T 51027c9d845e4cd48fff13aa352bf1d7 P 1e1d21a1e555413fbd33ae2b59528073: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 04:10:28.856695 19212 tablet_bootstrap.cc:492] T 51027c9d845e4cd48fff13aa352bf1d7 P 0b3bbbe7b27f410cab32a1123b24a7e2: Bootstrap starting.
I20260810 04:10:28.857359 19212 tablet_bootstrap.cc:654] T 51027c9d845e4cd48fff13aa352bf1d7 P 0b3bbbe7b27f410cab32a1123b24a7e2: Neither blocks nor log segments found. Creating new log.
I20260810 04:10:28.857739 19212 log.cc:824] T 51027c9d845e4cd48fff13aa352bf1d7 P 0b3bbbe7b27f410cab32a1123b24a7e2: Log is configured to *not* fsync() on all Append() calls
I20260810 04:10:28.858073 19213 tablet_bootstrap.cc:492] T 51027c9d845e4cd48fff13aa352bf1d7 P 7f10750c69d14b6d85496fee617cf5d0: Bootstrap starting.
I20260810 04:10:28.858446 19212 tablet_bootstrap.cc:492] T 51027c9d845e4cd48fff13aa352bf1d7 P 0b3bbbe7b27f410cab32a1123b24a7e2: No bootstrap required, opened a new log
I20260810 04:10:28.858532 19212 ts_tablet_manager.cc:1397] T 51027c9d845e4cd48fff13aa352bf1d7 P 0b3bbbe7b27f410cab32a1123b24a7e2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 04:10:28.858718 19213 tablet_bootstrap.cc:654] T 51027c9d845e4cd48fff13aa352bf1d7 P 7f10750c69d14b6d85496fee617cf5d0: Neither blocks nor log segments found. Creating new log.
I20260810 04:10:28.859040 19213 log.cc:824] T 51027c9d845e4cd48fff13aa352bf1d7 P 7f10750c69d14b6d85496fee617cf5d0: Log is configured to *not* fsync() on all Append() calls
I20260810 04:10:28.859411 19210 raft_consensus.cc:348] T 51027c9d845e4cd48fff13aa352bf1d7 P 1e1d21a1e555413fbd33ae2b59528073 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b3bbbe7b27f410cab32a1123b24a7e2" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 32937 } } peers { permanent_uuid: "1e1d21a1e555413fbd33ae2b59528073" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 37185 } } peers { permanent_uuid: "7f10750c69d14b6d85496fee617cf5d0" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 40239 } }
I20260810 04:10:28.859579 19210 raft_consensus.cc:374] T 51027c9d845e4cd48fff13aa352bf1d7 P 1e1d21a1e555413fbd33ae2b59528073 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:10:28.859628 19210 raft_consensus.cc:729] T 51027c9d845e4cd48fff13aa352bf1d7 P 1e1d21a1e555413fbd33ae2b59528073 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e1d21a1e555413fbd33ae2b59528073, State: Initialized, Role: FOLLOWER
I20260810 04:10:28.859846 19213 tablet_bootstrap.cc:492] T 51027c9d845e4cd48fff13aa352bf1d7 P 7f10750c69d14b6d85496fee617cf5d0: No bootstrap required, opened a new log
I20260810 04:10:28.859767 19210 consensus_queue.cc:260] T 51027c9d845e4cd48fff13aa352bf1d7 P 1e1d21a1e555413fbd33ae2b59528073 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b3bbbe7b27f410cab32a1123b24a7e2" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 32937 } } peers { permanent_uuid: "1e1d21a1e555413fbd33ae2b59528073" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 37185 } } peers { permanent_uuid: "7f10750c69d14b6d85496fee617cf5d0" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 40239 } }
I20260810 04:10:28.860019 19213 ts_tablet_manager.cc:1397] T 51027c9d845e4cd48fff13aa352bf1d7 P 7f10750c69d14b6d85496fee617cf5d0: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 04:10:28.860080 19210 ts_tablet_manager.cc:1428] T 51027c9d845e4cd48fff13aa352bf1d7 P 1e1d21a1e555413fbd33ae2b59528073: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260810 04:10:28.860235 19195 heartbeater.cc:503] Master 127.31.32.62:35141 was elected leader, sending a full tablet report...
I20260810 04:10:28.861441 19212 raft_consensus.cc:348] T 51027c9d845e4cd48fff13aa352bf1d7 P 0b3bbbe7b27f410cab32a1123b24a7e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b3bbbe7b27f410cab32a1123b24a7e2" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 32937 } } peers { permanent_uuid: "1e1d21a1e555413fbd33ae2b59528073" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 37185 } } peers { permanent_uuid: "7f10750c69d14b6d85496fee617cf5d0" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 40239 } }
I20260810 04:10:28.861605 19212 raft_consensus.cc:374] T 51027c9d845e4cd48fff13aa352bf1d7 P 0b3bbbe7b27f410cab32a1123b24a7e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:10:28.861639 19212 raft_consensus.cc:729] T 51027c9d845e4cd48fff13aa352bf1d7 P 0b3bbbe7b27f410cab32a1123b24a7e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b3bbbe7b27f410cab32a1123b24a7e2, State: Initialized, Role: FOLLOWER
I20260810 04:10:28.861763 19212 consensus_queue.cc:260] T 51027c9d845e4cd48fff13aa352bf1d7 P 0b3bbbe7b27f410cab32a1123b24a7e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b3bbbe7b27f410cab32a1123b24a7e2" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 32937 } } peers { permanent_uuid: "1e1d21a1e555413fbd33ae2b59528073" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 37185 } } peers { permanent_uuid: "7f10750c69d14b6d85496fee617cf5d0" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 40239 } }
I20260810 04:10:28.862023 19212 ts_tablet_manager.cc:1428] T 51027c9d845e4cd48fff13aa352bf1d7 P 0b3bbbe7b27f410cab32a1123b24a7e2: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 04:10:28.862083 18906 heartbeater.cc:503] Master 127.31.32.62:35141 was elected leader, sending a full tablet report...
I20260810 04:10:28.862447 19213 raft_consensus.cc:348] T 51027c9d845e4cd48fff13aa352bf1d7 P 7f10750c69d14b6d85496fee617cf5d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b3bbbe7b27f410cab32a1123b24a7e2" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 32937 } } peers { permanent_uuid: "1e1d21a1e555413fbd33ae2b59528073" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 37185 } } peers { permanent_uuid: "7f10750c69d14b6d85496fee617cf5d0" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 40239 } }
I20260810 04:10:28.862623 19213 raft_consensus.cc:374] T 51027c9d845e4cd48fff13aa352bf1d7 P 7f10750c69d14b6d85496fee617cf5d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:10:28.862670 19213 raft_consensus.cc:729] T 51027c9d845e4cd48fff13aa352bf1d7 P 7f10750c69d14b6d85496fee617cf5d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f10750c69d14b6d85496fee617cf5d0, State: Initialized, Role: FOLLOWER
I20260810 04:10:28.862804 19213 consensus_queue.cc:260] T 51027c9d845e4cd48fff13aa352bf1d7 P 7f10750c69d14b6d85496fee617cf5d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b3bbbe7b27f410cab32a1123b24a7e2" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 32937 } } peers { permanent_uuid: "1e1d21a1e555413fbd33ae2b59528073" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 37185 } } peers { permanent_uuid: "7f10750c69d14b6d85496fee617cf5d0" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 40239 } }
I20260810 04:10:28.863061 19213 ts_tablet_manager.cc:1428] T 51027c9d845e4cd48fff13aa352bf1d7 P 7f10750c69d14b6d85496fee617cf5d0: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260810 04:10:28.863135 19050 heartbeater.cc:503] Master 127.31.32.62:35141 was elected leader, sending a full tablet report...
I20260810 04:10:28.881906 19216 raft_consensus.cc:482] T 51027c9d845e4cd48fff13aa352bf1d7 P 1e1d21a1e555413fbd33ae2b59528073 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 04:10:28.882064 19216 raft_consensus.cc:504] T 51027c9d845e4cd48fff13aa352bf1d7 P 1e1d21a1e555413fbd33ae2b59528073 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b3bbbe7b27f410cab32a1123b24a7e2" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 32937 } } peers { permanent_uuid: "1e1d21a1e555413fbd33ae2b59528073" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 37185 } } peers { permanent_uuid: "7f10750c69d14b6d85496fee617cf5d0" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 40239 } }
I20260810 04:10:28.882463 19216 leader_election.cc:288] T 51027c9d845e4cd48fff13aa352bf1d7 P 1e1d21a1e555413fbd33ae2b59528073 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b3bbbe7b27f410cab32a1123b24a7e2 (127.31.32.1:32937), 7f10750c69d14b6d85496fee617cf5d0 (127.31.32.2:40239)
I20260810 04:10:28.885651 19197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:28.882614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.2:40239 (local address 127.31.32.3:50033)
0810 04:10:28.882681 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0810 04:10:28.882695 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 04:10:28.882742 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:10:28.882933 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:10:28.882934 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:10:28.882937 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:10:28.883004 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:10:28.883006 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:10:28.883740 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:10:28.883741 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:10:28.884793 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:10:28.884796 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:10:28.885384 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:10:28.885385 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:10:28.885459 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:10:28.885464 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 04:10:28.885514 (+ 50us) client_negotiation.cc:240] Negotiation successful
0810 04:10:28.885556 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260810 04:10:28.885664 19209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:28.882726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.3:50033 (local address 127.31.32.2:40239)
0810 04:10:28.882788 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0810 04:10:28.882790 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:10:28.882801 (+ 11us) server_negotiation.cc:407] Connection header received
0810 04:10:28.882838 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:10:28.882839 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:10:28.882848 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:10:28.882909 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:10:28.883081 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:28.883659 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:28.884881 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:28.885301 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:28.885357 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:10:28.885364 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0810 04:10:28.885532 (+ 168us) server_negotiation.cc:299] Negotiation successful
0810 04:10:28.885580 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260810 04:10:28.885739 19208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:28.882731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.3:39925 (local address 127.31.32.1:32937)
0810 04:10:28.882810 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0810 04:10:28.882813 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:10:28.883066 (+ 253us) server_negotiation.cc:407] Connection header received
0810 04:10:28.883126 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:10:28.883127 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:10:28.883139 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:10:28.883184 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:10:28.883339 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:28.883850 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:28.884980 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:28.885377 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:28.885449 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:10:28.885458 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0810 04:10:28.885624 (+ 166us) server_negotiation.cc:299] Negotiation successful
0810 04:10:28.885666 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 04:10:28.885982 19000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51027c9d845e4cd48fff13aa352bf1d7" candidate_uuid: "1e1d21a1e555413fbd33ae2b59528073" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f10750c69d14b6d85496fee617cf5d0" is_pre_election: true
I20260810 04:10:28.886107 19000 raft_consensus.cc:2454] T 51027c9d845e4cd48fff13aa352bf1d7 P 7f10750c69d14b6d85496fee617cf5d0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e1d21a1e555413fbd33ae2b59528073 in term 0.
I20260810 04:10:28.886224 19219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:28.882615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.1:32937 (local address 127.31.32.3:39925)
0810 04:10:28.882988 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0810 04:10:28.883004 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 04:10:28.883073 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:10:28.883196 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:10:28.883196 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:10:28.883198 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:10:28.883274 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:10:28.883276 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:10:28.883928 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:10:28.883929 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:10:28.884893 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:10:28.884896 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:10:28.885460 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:10:28.885461 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:10:28.885545 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:10:28.885549 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 04:10:28.885605 (+ 56us) client_negotiation.cc:240] Negotiation successful
0810 04:10:28.885638 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":355,"mutex_wait_us":140,"thread_start_us":158,"threads_started":1}
I20260810 04:10:28.886374 19071 leader_election.cc:302] T 51027c9d845e4cd48fff13aa352bf1d7 P 1e1d21a1e555413fbd33ae2b59528073 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e1d21a1e555413fbd33ae2b59528073, 7f10750c69d14b6d85496fee617cf5d0; no voters:
I20260810 04:10:28.886497 19216 raft_consensus.cc:2781] T 51027c9d845e4cd48fff13aa352bf1d7 P 1e1d21a1e555413fbd33ae2b59528073 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 04:10:28.886571 19216 raft_consensus.cc:482] T 51027c9d845e4cd48fff13aa352bf1d7 P 1e1d21a1e555413fbd33ae2b59528073 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 04:10:28.886596 19216 raft_consensus.cc:3037] T 51027c9d845e4cd48fff13aa352bf1d7 P 1e1d21a1e555413fbd33ae2b59528073 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:10:28.886547 18856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51027c9d845e4cd48fff13aa352bf1d7" candidate_uuid: "1e1d21a1e555413fbd33ae2b59528073" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b3bbbe7b27f410cab32a1123b24a7e2" is_pre_election: true
I20260810 04:10:28.886689 18856 raft_consensus.cc:2454] T 51027c9d845e4cd48fff13aa352bf1d7 P 0b3bbbe7b27f410cab32a1123b24a7e2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e1d21a1e555413fbd33ae2b59528073 in term 0.
I20260810 04:10:28.887390 19216 raft_consensus.cc:504] T 51027c9d845e4cd48fff13aa352bf1d7 P 1e1d21a1e555413fbd33ae2b59528073 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b3bbbe7b27f410cab32a1123b24a7e2" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 32937 } } peers { permanent_uuid: "1e1d21a1e555413fbd33ae2b59528073" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 37185 } } peers { permanent_uuid: "7f10750c69d14b6d85496fee617cf5d0" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 40239 } }
I20260810 04:10:28.887533 19216 leader_election.cc:288] T 51027c9d845e4cd48fff13aa352bf1d7 P 1e1d21a1e555413fbd33ae2b59528073 [CANDIDATE]: Term 1 election: Requested vote from peers 0b3bbbe7b27f410cab32a1123b24a7e2 (127.31.32.1:32937), 7f10750c69d14b6d85496fee617cf5d0 (127.31.32.2:40239)
I20260810 04:10:28.887717 18856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51027c9d845e4cd48fff13aa352bf1d7" candidate_uuid: "1e1d21a1e555413fbd33ae2b59528073" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b3bbbe7b27f410cab32a1123b24a7e2"
I20260810 04:10:28.887717 19000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51027c9d845e4cd48fff13aa352bf1d7" candidate_uuid: "1e1d21a1e555413fbd33ae2b59528073" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f10750c69d14b6d85496fee617cf5d0"
I20260810 04:10:28.887796 19000 raft_consensus.cc:3037] T 51027c9d845e4cd48fff13aa352bf1d7 P 7f10750c69d14b6d85496fee617cf5d0 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:10:28.887806 18856 raft_consensus.cc:3037] T 51027c9d845e4cd48fff13aa352bf1d7 P 0b3bbbe7b27f410cab32a1123b24a7e2 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:10:28.888782 18856 raft_consensus.cc:2454] T 51027c9d845e4cd48fff13aa352bf1d7 P 0b3bbbe7b27f410cab32a1123b24a7e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e1d21a1e555413fbd33ae2b59528073 in term 1.
I20260810 04:10:28.888782 19000 raft_consensus.cc:2454] T 51027c9d845e4cd48fff13aa352bf1d7 P 7f10750c69d14b6d85496fee617cf5d0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e1d21a1e555413fbd33ae2b59528073 in term 1.
I20260810 04:10:28.888967 19071 leader_election.cc:302] T 51027c9d845e4cd48fff13aa352bf1d7 P 1e1d21a1e555413fbd33ae2b59528073 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e1d21a1e555413fbd33ae2b59528073, 7f10750c69d14b6d85496fee617cf5d0; no voters:
I20260810 04:10:28.889165 19216 raft_consensus.cc:2781] T 51027c9d845e4cd48fff13aa352bf1d7 P 1e1d21a1e555413fbd33ae2b59528073 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:10:28.889302 19216 raft_consensus.cc:686] T 51027c9d845e4cd48fff13aa352bf1d7 P 1e1d21a1e555413fbd33ae2b59528073 [term 1 LEADER]: Becoming Leader. State: Replica: 1e1d21a1e555413fbd33ae2b59528073, State: Running, Role: LEADER
I20260810 04:10:28.889377 19216 consensus_queue.cc:237] T 51027c9d845e4cd48fff13aa352bf1d7 P 1e1d21a1e555413fbd33ae2b59528073 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b3bbbe7b27f410cab32a1123b24a7e2" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 32937 } } peers { permanent_uuid: "1e1d21a1e555413fbd33ae2b59528073" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 37185 } } peers { permanent_uuid: "7f10750c69d14b6d85496fee617cf5d0" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 40239 } }
I20260810 04:10:28.890236 18694 catalog_manager.cc:5495] T 51027c9d845e4cd48fff13aa352bf1d7 P 1e1d21a1e555413fbd33ae2b59528073 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1e1d21a1e555413fbd33ae2b59528073 (127.31.32.3). New cstate: current_term: 1 leader_uuid: "1e1d21a1e555413fbd33ae2b59528073" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b3bbbe7b27f410cab32a1123b24a7e2" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 32937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e1d21a1e555413fbd33ae2b59528073" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 37185 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7f10750c69d14b6d85496fee617cf5d0" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 40239 } health_report { overall_health: UNKNOWN } } }
I20260810 04:10:28.898368 19206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:28.895869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34600 (local address 127.31.32.3:37185)
0810 04:10:28.895911 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0810 04:10:28.895913 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:10:28.895939 (+ 26us) server_negotiation.cc:407] Connection header received
0810 04:10:28.895971 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:10:28.895972 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:10:28.895999 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:10:28.896074 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 04:10:28.896260 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:28.896750 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:28.897503 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:28.897730 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:28.897805 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:10:28.898007 (+ 202us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 04:10:28.898114 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 04:10:28.898200 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0810 04:10:28.898243 (+ 43us) server_negotiation.cc:299] Negotiation successful
0810 04:10:28.898261 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260810 04:10:28.900105 19000 raft_consensus.cc:1261] T 51027c9d845e4cd48fff13aa352bf1d7 P 7f10750c69d14b6d85496fee617cf5d0 [term 1 FOLLOWER]: Refusing update from remote peer 1e1d21a1e555413fbd33ae2b59528073: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:10:28.900108 18856 raft_consensus.cc:1261] T 51027c9d845e4cd48fff13aa352bf1d7 P 0b3bbbe7b27f410cab32a1123b24a7e2 [term 1 FOLLOWER]: Refusing update from remote peer 1e1d21a1e555413fbd33ae2b59528073: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:10:28.900506 19220 consensus_queue.cc:1035] T 51027c9d845e4cd48fff13aa352bf1d7 P 1e1d21a1e555413fbd33ae2b59528073 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f10750c69d14b6d85496fee617cf5d0" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 40239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:10:28.901160 19220 consensus_queue.cc:1035] T 51027c9d845e4cd48fff13aa352bf1d7 P 1e1d21a1e555413fbd33ae2b59528073 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b3bbbe7b27f410cab32a1123b24a7e2" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 32937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:10:28.902781 19223 mvcc.cc:205] Tried to move back new op lower bound from 7316828278372130816 to 7316828278331580416. Current Snapshot: MvccSnapshot[applied={T|T < 7316828278372130816}]
I20260810 04:10:28.905819 18695 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54350:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 04:10:28.905926 18695 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54350:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 04:10:28.906621 18695 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c5dffc7b12dd4ec582ede9ceac7385bb: Sending DeleteTablet for 3 replicas of tablet 51027c9d845e4cd48fff13aa352bf1d7
I20260810 04:10:28.906965 18832 tablet_service.cc:1514] Processing DeleteTablet for tablet 51027c9d845e4cd48fff13aa352bf1d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:10:28 UTC) from {username='slave'} at 127.0.0.1:45142
I20260810 04:10:28.906985 18978 tablet_service.cc:1514] Processing DeleteTablet for tablet 51027c9d845e4cd48fff13aa352bf1d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:10:28 UTC) from {username='slave'} at 127.0.0.1:47058
I20260810 04:10:28.907279 19120 tablet_service.cc:1514] Processing DeleteTablet for tablet 51027c9d845e4cd48fff13aa352bf1d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:10:28 UTC) from {username='slave'} at 127.0.0.1:34596
I20260810 04:10:28.907333 31872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskws89Io/build/release/bin/kudu with pid 18763
I20260810 04:10:28.912258 19235 tablet_replica.cc:330] T 51027c9d845e4cd48fff13aa352bf1d7 P 1e1d21a1e555413fbd33ae2b59528073: stopping tablet replica
I20260810 04:10:28.912390 19235 raft_consensus.cc:2229] T 51027c9d845e4cd48fff13aa352bf1d7 P 1e1d21a1e555413fbd33ae2b59528073 [term 1 LEADER]: Raft consensus shutting down.
I20260810 04:10:28.912536 19235 raft_consensus.cc:2258] T 51027c9d845e4cd48fff13aa352bf1d7 P 1e1d21a1e555413fbd33ae2b59528073 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:10:28.912923 19235 ts_tablet_manager.cc:1905] T 51027c9d845e4cd48fff13aa352bf1d7 P 1e1d21a1e555413fbd33ae2b59528073: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:10:28.914623 19235 ts_tablet_manager.cc:1918] T 51027c9d845e4cd48fff13aa352bf1d7 P 1e1d21a1e555413fbd33ae2b59528073: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 04:10:28.914709 19235 log.cc:1196] T 51027c9d845e4cd48fff13aa352bf1d7 P 1e1d21a1e555413fbd33ae2b59528073: Deleting WAL directory at /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786334794676589-31872-0/minicluster-data/ts-2/wal/wals/51027c9d845e4cd48fff13aa352bf1d7
I20260810 04:10:28.914974 19235 ts_tablet_manager.cc:1939] T 51027c9d845e4cd48fff13aa352bf1d7 P 1e1d21a1e555413fbd33ae2b59528073: Deleting consensus metadata
I20260810 04:10:28.915405 18678 catalog_manager.cc:4841] TS 1e1d21a1e555413fbd33ae2b59528073 (127.31.32.3:37185): tablet 51027c9d845e4cd48fff13aa352bf1d7 (table test-table [id=326d13f07aed4c7eb6e848a39aa18cea]) successfully deleted
W20260810 04:10:28.916366 18676 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.32.1:32937 (error 108)
I20260810 04:10:28.916872 19204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:28.916737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.1:32937 (local address 127.0.0.1:45146)
0810 04:10:28.916792 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0810 04:10:28.916834 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.1:32937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260810 04:10:28.916958 31872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskws89Io/build/release/bin/kudu with pid 18909
W20260810 04:10:28.917070 18676 catalog_manager.cc:4584] TS 0b3bbbe7b27f410cab32a1123b24a7e2 (127.31.32.1:32937): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 51027c9d845e4cd48fff13aa352bf1d7: Network error: Client connection negotiation failed: client connection to 127.31.32.1:32937: connect: Connection refused (error 111)
I20260810 04:10:28.931954 31872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskws89Io/build/release/bin/kudu with pid 19053
I20260810 04:10:28.932000 19204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:28.931848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.2:40239 (local address 127.0.0.1:47068)
0810 04:10:28.931923 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0810 04:10:28.931959 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.2:40239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260810 04:10:28.938330 31872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskws89Io/build/release/bin/kudu with pid 18660
2026-08-10T04:10:28Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (713 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786334794676589-31872-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 04:10:28 dist-test-slave-8w9v krb5kdc[19239](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 04:10:28 dist-test-slave-8w9v krb5kdc[19239](info): set up 2 sockets
Aug 10 04:10:28 dist-test-slave-8w9v krb5kdc[19239](info): commencing operation
krb5kdc: starting...
W20260810 04:10:30.043009 31872 mini_kdc.cc:121] Time spent starting KDC: real 1.084s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786334794676589-31872-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786334794676589-31872-0/krb5kdc/test-user.keytab.
Aug 10 04:10:30 dist-test-slave-8w9v krb5kdc[19239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335030, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T04:10:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T04:10:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.32.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.32.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.32.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.62.keytab.
Entry for principal kudu/127.31.32.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.62.keytab.
I20260810 04:10:30.171098 31872 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskws89Io/build/release/bin/kudu
/tmp/dist-test-taskws89Io/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786334794676589-31872-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786334794676589-31872-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786334794676589-31872-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786334794676589-31872-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.32.62:41205
--webserver_interface=127.31.32.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.62.keytab
--principal=kudu/127.31.32.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.32.20:38525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.32.62:41205
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786334794676589-31872-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786334794676589-31872-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786334794676589-31872-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:10:30.255606 19253 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:10:30.255863 19253 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:10:30.255908 19253 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:10:30.257656 19253 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:10:30.257745 19253 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:10:30.257772 19253 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:10:30.257797 19253 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:10:30.257831 19253 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:10:30.259369 19253 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.32.20:38525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786334794676589-31872-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786334794676589-31872-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.32.62:41205
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.62.keytab
--principal=kudu/127.31.32.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.32.62:41205
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786334794676589-31872-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.32.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19253
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786334794676589-31872-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:10:30.259661 19253 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:10:30.259907 19253 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:10:30.264863 19259 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:10:30.265043 19253 server_base.cc:1034] running on GCE node
W20260810 04:10:30.264921 19258 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:10:30.264863 19261 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:10:30.265395 19253 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:10:30.265705 19253 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:10:30.266898 19253 hybrid_clock.cc:648] HybridClock initialized: now 1786335030266862 us; error 31 us; skew 500 ppm
Aug 10 04:10:30 dist-test-slave-8w9v krb5kdc[19239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335030, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:10:30.269721 19253 init.cc:377] Logged in from keytab as kudu/127.31.32.62@KRBTEST.COM (short username kudu)
I20260810 04:10:30.270150 19253 webserver.cc:460] Webserver started at http://127.31.32.62:37387/ using document root <none> and password file <none>
I20260810 04:10:30.270431 19253 fs_manager.cc:359] Metadata directory not provided
I20260810 04:10:30.270514 19253 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:10:30.270665 19253 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:10:30.271724 19253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786334794676589-31872-0/minicluster-data/master-0/data/instance:
uuid: "5c052a9a931d4736b3f86a80ee809941"
format_stamp: "Formatted at 2026-08-10 04:10:30 on dist-test-slave-8w9v"
server_key: "bb7f8bb5143461e29c1147886c639834"
server_key_iv: "22d0996c0c7288f7e37d874499e0f6a1"
server_key_version: "encryptionkey@0"
I20260810 04:10:30.272100 19253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786334794676589-31872-0/minicluster-data/master-0/wal/instance:
uuid: "5c052a9a931d4736b3f86a80ee809941"
format_stamp: "Formatted at 2026-08-10 04:10:30 on dist-test-slave-8w9v"
server_key: "bb7f8bb5143461e29c1147886c639834"
server_key_iv: "22d0996c0c7288f7e37d874499e0f6a1"
server_key_version: "encryptionkey@0"
I20260810 04:10:30.273518 19253 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:10:30.274325 19268 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:10:30.274521 19253 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:10:30.274631 19253 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786334794676589-31872-0/minicluster-data/master-0/data,/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786334794676589-31872-0/minicluster-data/master-0/wal
uuid: "5c052a9a931d4736b3f86a80ee809941"
format_stamp: "Formatted at 2026-08-10 04:10:30 on dist-test-slave-8w9v"
server_key: "bb7f8bb5143461e29c1147886c639834"
server_key_iv: "22d0996c0c7288f7e37d874499e0f6a1"
server_key_version: "encryptionkey@0"
I20260810 04:10:30.274709 19253 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786334794676589-31872-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786334794676589-31872-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786334794676589-31872-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:10:30.287736 19253 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:10:30.289744 19253 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:10:30.289912 19253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:10:30.294023 19253 rpc_server.cc:307] RPC server started. Bound to: 127.31.32.62:41205
I20260810 04:10:30.294075 19328 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.32.62:41205 every 8 connection(s)
I20260810 04:10:30.294423 19253 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786334794676589-31872-0/minicluster-data/master-0/data/info.pb
I20260810 04:10:30.295341 19329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:10:30.296403 31872 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskws89Io/build/release/bin/kudu as pid 19253
I20260810 04:10:30.296490 31872 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786334794676589-31872-0/minicluster-data/master-0/wal/instance
I20260810 04:10:30.296680 31872 external_mini_cluster.cc:1428] Setting key 9155a19f3e1e4bc8b63b6da24649b21e
I20260810 04:10:30.298431 19329 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c052a9a931d4736b3f86a80ee809941: Bootstrap starting.
I20260810 04:10:30.299127 19329 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5c052a9a931d4736b3f86a80ee809941: Neither blocks nor log segments found. Creating new log.
I20260810 04:10:30.299491 19329 log.cc:824] T 00000000000000000000000000000000 P 5c052a9a931d4736b3f86a80ee809941: Log is configured to *not* fsync() on all Append() calls
I20260810 04:10:30.300274 19329 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c052a9a931d4736b3f86a80ee809941: No bootstrap required, opened a new log
I20260810 04:10:30.302630 19329 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5c052a9a931d4736b3f86a80ee809941 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c052a9a931d4736b3f86a80ee809941" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 41205 } }
I20260810 04:10:30.302771 19329 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5c052a9a931d4736b3f86a80ee809941 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:10:30.302794 19329 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5c052a9a931d4736b3f86a80ee809941 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c052a9a931d4736b3f86a80ee809941, State: Initialized, Role: FOLLOWER
I20260810 04:10:30.302927 19329 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5c052a9a931d4736b3f86a80ee809941 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c052a9a931d4736b3f86a80ee809941" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 41205 } }
I20260810 04:10:30.303031 19329 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5c052a9a931d4736b3f86a80ee809941 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 04:10:30.303077 19329 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5c052a9a931d4736b3f86a80ee809941 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 04:10:30.303146 19329 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5c052a9a931d4736b3f86a80ee809941 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:10:30.303710 19329 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5c052a9a931d4736b3f86a80ee809941 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c052a9a931d4736b3f86a80ee809941" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 41205 } }
I20260810 04:10:30.303834 19329 leader_election.cc:302] T 00000000000000000000000000000000 P 5c052a9a931d4736b3f86a80ee809941 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5c052a9a931d4736b3f86a80ee809941; no voters:
I20260810 04:10:30.304056 19329 leader_election.cc:288] T 00000000000000000000000000000000 P 5c052a9a931d4736b3f86a80ee809941 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 04:10:30.304090 19334 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5c052a9a931d4736b3f86a80ee809941 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:10:30.304239 19334 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5c052a9a931d4736b3f86a80ee809941 [term 1 LEADER]: Becoming Leader. State: Replica: 5c052a9a931d4736b3f86a80ee809941, State: Running, Role: LEADER
I20260810 04:10:30.304323 19334 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5c052a9a931d4736b3f86a80ee809941 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c052a9a931d4736b3f86a80ee809941" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 41205 } }
Aug 10 04:10:30 dist-test-slave-8w9v krb5kdc[19239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335030, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.32.62@KRBTEST.COM
I20260810 04:10:30.304358 19329 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5c052a9a931d4736b3f86a80ee809941 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:10:30.304733 19335 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c052a9a931d4736b3f86a80ee809941 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5c052a9a931d4736b3f86a80ee809941" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c052a9a931d4736b3f86a80ee809941" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 41205 } } }
I20260810 04:10:30.304823 19335 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c052a9a931d4736b3f86a80ee809941 [sys.catalog]: This master's current role is: LEADER
I20260810 04:10:30.305080 19342 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 04:10:30.305972 19342 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 04:10:30.305975 19336 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c052a9a931d4736b3f86a80ee809941 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5c052a9a931d4736b3f86a80ee809941. Latest consensus state: current_term: 1 leader_uuid: "5c052a9a931d4736b3f86a80ee809941" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c052a9a931d4736b3f86a80ee809941" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 41205 } } }
I20260810 04:10:30.308313 19336 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c052a9a931d4736b3f86a80ee809941 [sys.catalog]: This master's current role is: LEADER
I20260810 04:10:30.308650 19332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:30.297276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53990 (local address 127.31.32.62:41205)
0810 04:10:30.297524 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0810 04:10:30.297534 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 04:10:30.297572 (+ 38us) server_negotiation.cc:407] Connection header received
0810 04:10:30.298101 (+ 529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:10:30.298110 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:10:30.298176 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:10:30.298335 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:10:30.299043 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:30.300466 (+ 1423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:30.301241 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:30.301880 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:30.302019 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:10:30.304969 (+ 2950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:10:30.304973 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:10:30.304974 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:10:30.304987 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:10:30.306981 (+ 1994us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:10:30.307405 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:10:30.307407 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:10:30.307407 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:10:30.307502 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:10:30.307734 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:10:30.307735 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:10:30.307735 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:10:30.307957 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:10:30.308067 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0810 04:10:30.308272 (+ 205us) server_negotiation.cc:299] Negotiation successful
0810 04:10:30.308434 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":197,"thread_start_us":75,"threads_started":1}
I20260810 04:10:30.309582 19342 catalog_manager.cc:1347] Generated new cluster ID: a666e2ed04404541abda5f265a82ddc7
I20260810 04:10:30.309624 19342 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 04:10:30.323000 19342 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 04:10:30.323695 19342 catalog_manager.cc:1504] Loading token signing keys...
I20260810 04:10:30.328380 19342 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5c052a9a931d4736b3f86a80ee809941: Generated new TSK 0
I20260810 04:10:30.328573 19342 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.32.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.32.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.32.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.1.keytab.
Entry for principal kudu/127.31.32.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.1.keytab.
I20260810 04:10:30.356769 31872 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskws89Io/build/release/bin/kudu
/tmp/dist-test-taskws89Io/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786334794676589-31872-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786334794676589-31872-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786334794676589-31872-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786334794676589-31872-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.32.1:0
--local_ip_for_outbound_sockets=127.31.32.1
--webserver_interface=127.31.32.1
--webserver_port=0
--tserver_master_addrs=127.31.32.62:41205
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.1.keytab
--principal=kudu/127.31.32.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.32.20:38525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786334794676589-31872-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786334794676589-31872-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786334794676589-31872-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:10:30.441135 19358 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:10:30.441371 19358 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:10:30.441411 19358 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:10:30.442596 19358 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:10:30.442659 19358 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:10:30.442704 19358 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.32.1
I20260810 04:10:30.444267 19358 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.32.20:38525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786334794676589-31872-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786334794676589-31872-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.1.keytab
--principal=kudu/127.31.32.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.32.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786334794676589-31872-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.32.1
--webserver_port=0
--tserver_master_addrs=127.31.32.62:41205
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19358
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.32.1
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786334794676589-31872-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:10:30.444548 19358 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:10:30.444782 19358 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:10:30.449822 19366 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:10:30.449808 19364 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:10:30.449791 19363 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:10:30.450065 19358 server_base.cc:1034] running on GCE node
I20260810 04:10:30.450253 19358 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:10:30.450495 19358 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:10:30.451637 19358 hybrid_clock.cc:648] HybridClock initialized: now 1786335030451623 us; error 28 us; skew 500 ppm
Aug 10 04:10:30 dist-test-slave-8w9v krb5kdc[19239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335030, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:10:30.454180 19358 init.cc:377] Logged in from keytab as kudu/127.31.32.1@KRBTEST.COM (short username kudu)
I20260810 04:10:30.454640 19358 webserver.cc:460] Webserver started at http://127.31.32.1:32777/ using document root <none> and password file <none>
I20260810 04:10:30.454906 19358 fs_manager.cc:359] Metadata directory not provided
I20260810 04:10:30.454974 19358 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:10:30.455119 19358 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:10:30.456255 19358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786334794676589-31872-0/minicluster-data/ts-0/data/instance:
uuid: "7dbbd9270f9e433d8fd19411edf43d81"
format_stamp: "Formatted at 2026-08-10 04:10:30 on dist-test-slave-8w9v"
server_key: "ac23611ee72a6ef5f0d35eb2f8718462"
server_key_iv: "0a421e37dbe9e07396f0eb628334cd7c"
server_key_version: "encryptionkey@0"
I20260810 04:10:30.456624 19358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786334794676589-31872-0/minicluster-data/ts-0/wal/instance:
uuid: "7dbbd9270f9e433d8fd19411edf43d81"
format_stamp: "Formatted at 2026-08-10 04:10:30 on dist-test-slave-8w9v"
server_key: "ac23611ee72a6ef5f0d35eb2f8718462"
server_key_iv: "0a421e37dbe9e07396f0eb628334cd7c"
server_key_version: "encryptionkey@0"
I20260810 04:10:30.458029 19358 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 04:10:30.458765 19374 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:10:30.458941 19358 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 04:10:30.459048 19358 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786334794676589-31872-0/minicluster-data/ts-0/data,/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786334794676589-31872-0/minicluster-data/ts-0/wal
uuid: "7dbbd9270f9e433d8fd19411edf43d81"
format_stamp: "Formatted at 2026-08-10 04:10:30 on dist-test-slave-8w9v"
server_key: "ac23611ee72a6ef5f0d35eb2f8718462"
server_key_iv: "0a421e37dbe9e07396f0eb628334cd7c"
server_key_version: "encryptionkey@0"
I20260810 04:10:30.459120 19358 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786334794676589-31872-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786334794676589-31872-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786334794676589-31872-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:10:30.478200 19358 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:10:30.480463 19358 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:10:30.480660 19358 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:10:30.480904 19358 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:10:30.481240 19358 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:10:30.481276 19358 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:10:30.481333 19358 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:10:30.481360 19358 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:10:30.486930 19358 rpc_server.cc:307] RPC server started. Bound to: 127.31.32.1:40765
I20260810 04:10:30.487013 19498 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.32.1:40765 every 8 connection(s)
I20260810 04:10:30.487351 19358 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786334794676589-31872-0/minicluster-data/ts-0/data/info.pb
I20260810 04:10:30.492422 31872 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskws89Io/build/release/bin/kudu as pid 19358
I20260810 04:10:30.492578 31872 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786334794676589-31872-0/minicluster-data/ts-0/wal/instance
I20260810 04:10:30.492792 31872 external_mini_cluster.cc:1428] Setting key 86094b34cd0044dfdaf97498d25bae48
Aug 10 04:10:30 dist-test-slave-8w9v krb5kdc[19239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335030, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.1@KRBTEST.COM for kudu/127.31.32.62@KRBTEST.COM
I20260810 04:10:30.497469 19332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:30.488178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.1:47543 (local address 127.31.32.62:41205)
0810 04:10:30.488225 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0810 04:10:30.488227 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:10:30.488444 (+ 217us) server_negotiation.cc:407] Connection header received
0810 04:10:30.488982 (+ 538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:10:30.488984 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:10:30.488998 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:10:30.489040 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:10:30.490004 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:30.490512 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:30.491521 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:30.491731 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:30.491817 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:10:30.494504 (+ 2687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:10:30.494506 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:10:30.494507 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:10:30.494515 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:10:30.496045 (+ 1530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:10:30.496495 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:10:30.496497 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:10:30.496497 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:10:30.496556 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:10:30.496812 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:10:30.496813 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:10:30.496814 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:10:30.496932 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:10:30.497002 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0810 04:10:30.497289 (+ 287us) server_negotiation.cc:299] Negotiation successful
0810 04:10:30.497381 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260810 04:10:30.497674 19501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:30.488159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:41205 (local address 127.31.32.1:47543)
0810 04:10:30.488317 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0810 04:10:30.488358 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 04:10:30.488862 (+ 504us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:10:30.489125 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:10:30.489128 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:10:30.489707 (+ 579us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:10:30.489899 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:10:30.489904 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:10:30.490618 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:10:30.490619 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:10:30.491428 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:10:30.491430 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:10:30.491830 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:10:30.491831 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:10:30.491958 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:10:30.492547 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:10:30.492558 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:10:30.494398 (+ 1840us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:10:30.496151 (+ 1753us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:10:30.496159 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:10:30.496161 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:10:30.496401 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:10:30.496624 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:10:30.496625 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:10:30.496626 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:10:30.496738 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:10:30.497006 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:10:30.497013 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:10:30.497150 (+ 137us) client_negotiation.cc:769] Sending connection context
0810 04:10:30.497251 (+ 101us) client_negotiation.cc:240] Negotiation successful
0810 04:10:30.497418 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":79,"threads_started":1}
I20260810 04:10:30.498250 19499 heartbeater.cc:347] Connected to a master server at 127.31.32.62:41205
I20260810 04:10:30.498395 19499 heartbeater.cc:464] Registering TS with master...
I20260810 04:10:30.498586 19499 heartbeater.cc:511] Master 127.31.32.62:41205 requested a full tablet report, sending...
I20260810 04:10:30.499183 19290 ts_manager.cc:194] Registered new tserver with Master: 7dbbd9270f9e433d8fd19411edf43d81 (127.31.32.1:40765)
I20260810 04:10:30.500208 19290 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.32.1@KRBTEST.COM'} at 127.31.32.1:47543
WARNING: no policy specified for kudu/127.31.32.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.32.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.32.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.2.keytab.
Entry for principal kudu/127.31.32.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.2.keytab.
I20260810 04:10:30.518972 31872 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskws89Io/build/release/bin/kudu
/tmp/dist-test-taskws89Io/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786334794676589-31872-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786334794676589-31872-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786334794676589-31872-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786334794676589-31872-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.32.2:0
--local_ip_for_outbound_sockets=127.31.32.2
--webserver_interface=127.31.32.2
--webserver_port=0
--tserver_master_addrs=127.31.32.62:41205
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.2.keytab
--principal=kudu/127.31.32.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.32.20:38525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786334794676589-31872-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786334794676589-31872-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786334794676589-31872-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:10:30.603786 19504 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:10:30.604065 19504 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:10:30.604154 19504 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:10:30.605631 19504 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:10:30.605690 19504 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:10:30.605748 19504 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.32.2
I20260810 04:10:30.607292 19504 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.32.20:38525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786334794676589-31872-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786334794676589-31872-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.2.keytab
--principal=kudu/127.31.32.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.32.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786334794676589-31872-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.32.2
--webserver_port=0
--tserver_master_addrs=127.31.32.62:41205
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19504
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.32.2
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786334794676589-31872-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:10:30.607545 19504 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:10:30.607815 19504 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:10:30.612993 19512 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:10:30.612993 19510 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:10:30.613256 19509 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:10:30.613258 19504 server_base.cc:1034] running on GCE node
I20260810 04:10:30.613533 19504 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:10:30.613778 19504 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:10:30.614936 19504 hybrid_clock.cc:648] HybridClock initialized: now 1786335030614928 us; error 29 us; skew 500 ppm
Aug 10 04:10:30 dist-test-slave-8w9v krb5kdc[19239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335030, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:10:30.617929 19504 init.cc:377] Logged in from keytab as kudu/127.31.32.2@KRBTEST.COM (short username kudu)
I20260810 04:10:30.618412 19504 webserver.cc:460] Webserver started at http://127.31.32.2:40261/ using document root <none> and password file <none>
I20260810 04:10:30.618654 19504 fs_manager.cc:359] Metadata directory not provided
I20260810 04:10:30.618700 19504 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:10:30.618873 19504 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:10:30.619927 19504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786334794676589-31872-0/minicluster-data/ts-1/data/instance:
uuid: "b09b28d0d79a4711ad09c8c43786741b"
format_stamp: "Formatted at 2026-08-10 04:10:30 on dist-test-slave-8w9v"
server_key: "56562f7b1900eca63073b57a46248e52"
server_key_iv: "ed922da92dc7ca83fa1f6530770fe46a"
server_key_version: "encryptionkey@0"
I20260810 04:10:30.620323 19504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786334794676589-31872-0/minicluster-data/ts-1/wal/instance:
uuid: "b09b28d0d79a4711ad09c8c43786741b"
format_stamp: "Formatted at 2026-08-10 04:10:30 on dist-test-slave-8w9v"
server_key: "56562f7b1900eca63073b57a46248e52"
server_key_iv: "ed922da92dc7ca83fa1f6530770fe46a"
server_key_version: "encryptionkey@0"
I20260810 04:10:30.621690 19504 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260810 04:10:30.622490 19519 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:10:30.622710 19504 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260810 04:10:30.622797 19504 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786334794676589-31872-0/minicluster-data/ts-1/data,/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786334794676589-31872-0/minicluster-data/ts-1/wal
uuid: "b09b28d0d79a4711ad09c8c43786741b"
format_stamp: "Formatted at 2026-08-10 04:10:30 on dist-test-slave-8w9v"
server_key: "56562f7b1900eca63073b57a46248e52"
server_key_iv: "ed922da92dc7ca83fa1f6530770fe46a"
server_key_version: "encryptionkey@0"
I20260810 04:10:30.622885 19504 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786334794676589-31872-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786334794676589-31872-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786334794676589-31872-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:10:30.646184 19504 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:10:30.648716 19504 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:10:30.648875 19504 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:10:30.649147 19504 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:10:30.649492 19504 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:10:30.649546 19504 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:10:30.649585 19504 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:10:30.649611 19504 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:10:30.655476 19504 rpc_server.cc:307] RPC server started. Bound to: 127.31.32.2:34333
I20260810 04:10:30.655525 19642 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.32.2:34333 every 8 connection(s)
I20260810 04:10:30.655865 19504 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786334794676589-31872-0/minicluster-data/ts-1/data/info.pb
Aug 10 04:10:30 dist-test-slave-8w9v krb5kdc[19239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335030, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.2@KRBTEST.COM for kudu/127.31.32.62@KRBTEST.COM
I20260810 04:10:30.664393 31872 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskws89Io/build/release/bin/kudu as pid 19504
I20260810 04:10:30.664516 31872 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786334794676589-31872-0/minicluster-data/ts-1/wal/instance
I20260810 04:10:30.664793 31872 external_mini_cluster.cc:1428] Setting key 7c7c0551332ac68c1a599f506c0ea478
I20260810 04:10:30.665238 19332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:30.656627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.2:34997 (local address 127.31.32.62:41205)
0810 04:10:30.656691 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0810 04:10:30.656694 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:10:30.656907 (+ 213us) server_negotiation.cc:407] Connection header received
0810 04:10:30.657385 (+ 478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:10:30.657386 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:10:30.657396 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:10:30.657454 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:10:30.658386 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:30.658874 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:30.659880 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:30.660074 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:30.660220 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:10:30.662252 (+ 2032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:10:30.662253 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:10:30.662253 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:10:30.662259 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:10:30.663702 (+ 1443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:10:30.664105 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:10:30.664106 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:10:30.664106 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:10:30.664207 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:10:30.664520 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:10:30.664521 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:10:30.664522 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:10:30.664662 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:10:30.664718 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0810 04:10:30.665072 (+ 354us) server_negotiation.cc:299] Negotiation successful
0810 04:10:30.665159 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260810 04:10:30.665330 19645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:30.656603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:41205 (local address 127.31.32.2:34997)
0810 04:10:30.656772 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0810 04:10:30.656814 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 04:10:30.657303 (+ 489us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:10:30.657524 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:10:30.657527 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:10:30.658096 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:10:30.658306 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:10:30.658310 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:10:30.658958 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:10:30.658959 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:10:30.659747 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:10:30.659749 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:10:30.660242 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:10:30.660243 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:10:30.660348 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:10:30.660779 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:10:30.660787 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:10:30.662170 (+ 1383us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:10:30.663803 (+ 1633us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:10:30.663810 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:10:30.663812 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:10:30.664034 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:10:30.664278 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:10:30.664279 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:10:30.664280 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:10:30.664431 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:10:30.664729 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:10:30.664736 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:10:30.664913 (+ 177us) client_negotiation.cc:769] Sending connection context
0810 04:10:30.665026 (+ 113us) client_negotiation.cc:240] Negotiation successful
0810 04:10:30.665147 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":89,"threads_started":1}
I20260810 04:10:30.665788 19643 heartbeater.cc:347] Connected to a master server at 127.31.32.62:41205
I20260810 04:10:30.665899 19643 heartbeater.cc:464] Registering TS with master...
I20260810 04:10:30.666086 19643 heartbeater.cc:511] Master 127.31.32.62:41205 requested a full tablet report, sending...
I20260810 04:10:30.666438 19290 ts_manager.cc:194] Registered new tserver with Master: b09b28d0d79a4711ad09c8c43786741b (127.31.32.2:34333)
I20260810 04:10:30.666821 19290 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.32.2@KRBTEST.COM'} at 127.31.32.2:34997
WARNING: no policy specified for kudu/127.31.32.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.32.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.32.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.3.keytab.
Entry for principal kudu/127.31.32.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.3.keytab.
I20260810 04:10:30.691318 31872 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskws89Io/build/release/bin/kudu
/tmp/dist-test-taskws89Io/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786334794676589-31872-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786334794676589-31872-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786334794676589-31872-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786334794676589-31872-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.32.3:0
--local_ip_for_outbound_sockets=127.31.32.3
--webserver_interface=127.31.32.3
--webserver_port=0
--tserver_master_addrs=127.31.32.62:41205
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.3.keytab
--principal=kudu/127.31.32.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.32.20:38525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786334794676589-31872-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786334794676589-31872-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786334794676589-31872-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:10:30.775868 19648 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:10:30.776088 19648 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:10:30.776113 19648 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:10:30.777344 19648 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:10:30.777396 19648 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:10:30.777424 19648 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.32.3
I20260810 04:10:30.778908 19648 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.32.20:38525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786334794676589-31872-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786334794676589-31872-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.3.keytab
--principal=kudu/127.31.32.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.32.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786334794676589-31872-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.32.3
--webserver_port=0
--tserver_master_addrs=127.31.32.62:41205
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19648
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.32.3
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786334794676589-31872-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:10:30.779107 19648 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:10:30.779289 19648 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:10:30.784490 19654 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:10:30.784497 19653 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:10:30.784461 19656 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:10:30.784722 19648 server_base.cc:1034] running on GCE node
I20260810 04:10:30.784945 19648 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:10:30.785195 19648 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:10:30.786351 19648 hybrid_clock.cc:648] HybridClock initialized: now 1786335030786341 us; error 25 us; skew 500 ppm
Aug 10 04:10:30 dist-test-slave-8w9v krb5kdc[19239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335030, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:10:30.788852 19648 init.cc:377] Logged in from keytab as kudu/127.31.32.3@KRBTEST.COM (short username kudu)
I20260810 04:10:30.789348 19648 webserver.cc:460] Webserver started at http://127.31.32.3:43165/ using document root <none> and password file <none>
I20260810 04:10:30.789600 19648 fs_manager.cc:359] Metadata directory not provided
I20260810 04:10:30.789669 19648 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:10:30.789816 19648 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:10:30.790886 19648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786334794676589-31872-0/minicluster-data/ts-2/data/instance:
uuid: "afce3cfcf56347aabad5114eb7614802"
format_stamp: "Formatted at 2026-08-10 04:10:30 on dist-test-slave-8w9v"
server_key: "e618e9f9f0d28da9252f80629ee4f10c"
server_key_iv: "8870102ae07613e80389969c2444542c"
server_key_version: "encryptionkey@0"
I20260810 04:10:30.791249 19648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786334794676589-31872-0/minicluster-data/ts-2/wal/instance:
uuid: "afce3cfcf56347aabad5114eb7614802"
format_stamp: "Formatted at 2026-08-10 04:10:30 on dist-test-slave-8w9v"
server_key: "e618e9f9f0d28da9252f80629ee4f10c"
server_key_iv: "8870102ae07613e80389969c2444542c"
server_key_version: "encryptionkey@0"
I20260810 04:10:30.792604 19648 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260810 04:10:30.793304 19663 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:10:30.793488 19648 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 04:10:30.793593 19648 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786334794676589-31872-0/minicluster-data/ts-2/data,/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786334794676589-31872-0/minicluster-data/ts-2/wal
uuid: "afce3cfcf56347aabad5114eb7614802"
format_stamp: "Formatted at 2026-08-10 04:10:30 on dist-test-slave-8w9v"
server_key: "e618e9f9f0d28da9252f80629ee4f10c"
server_key_iv: "8870102ae07613e80389969c2444542c"
server_key_version: "encryptionkey@0"
I20260810 04:10:30.793665 19648 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786334794676589-31872-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786334794676589-31872-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786334794676589-31872-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:10:30.819577 19648 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:10:30.821584 19648 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:10:30.821775 19648 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:10:30.822027 19648 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:10:30.822350 19648 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:10:30.822405 19648 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:10:30.822450 19648 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:10:30.822475 19648 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:10:30.828938 19648 rpc_server.cc:307] RPC server started. Bound to: 127.31.32.3:40159
I20260810 04:10:30.828976 19788 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.32.3:40159 every 8 connection(s)
I20260810 04:10:30.829344 19648 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786334794676589-31872-0/minicluster-data/ts-2/data/info.pb
Aug 10 04:10:30 dist-test-slave-8w9v krb5kdc[19239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335030, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.3@KRBTEST.COM for kudu/127.31.32.62@KRBTEST.COM
I20260810 04:10:30.836897 31872 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskws89Io/build/release/bin/kudu as pid 19648
I20260810 04:10:30.836994 31872 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786334794676589-31872-0/minicluster-data/ts-2/wal/instance
I20260810 04:10:30.837186 31872 external_mini_cluster.cc:1428] Setting key cc32c3d3daf8a7830f05aa48b4cedb26
I20260810 04:10:30.838686 19332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:30.830086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.3:39345 (local address 127.31.32.62:41205)
0810 04:10:30.830172 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0810 04:10:30.830174 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:10:30.830373 (+ 199us) server_negotiation.cc:407] Connection header received
0810 04:10:30.830852 (+ 479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:10:30.830853 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:10:30.830863 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:10:30.830916 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:10:30.831844 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:30.832435 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:30.833405 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:30.833584 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:30.833637 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:10:30.835628 (+ 1991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:10:30.835629 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:10:30.835630 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:10:30.835635 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:10:30.837355 (+ 1720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:10:30.837792 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:10:30.837793 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:10:30.837794 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:10:30.837856 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:10:30.838145 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:10:30.838145 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:10:30.838146 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:10:30.838273 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:10:30.838345 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0810 04:10:30.838568 (+ 223us) server_negotiation.cc:299] Negotiation successful
0810 04:10:30.838624 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 04:10:30.838925 19791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:30.830066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:41205 (local address 127.31.32.3:39345)
0810 04:10:30.830239 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0810 04:10:30.830283 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 04:10:30.830768 (+ 485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:10:30.830982 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:10:30.830985 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:10:30.831573 (+ 588us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:10:30.831765 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:10:30.831771 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:10:30.832514 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:10:30.832515 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:10:30.833325 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:10:30.833327 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:10:30.833658 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:10:30.833659 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:10:30.833758 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:10:30.834211 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:10:30.834218 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:10:30.835547 (+ 1329us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:10:30.837437 (+ 1890us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:10:30.837444 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:10:30.837446 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:10:30.837720 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:10:30.837948 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:10:30.837949 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:10:30.837949 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:10:30.838078 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:10:30.838347 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:10:30.838352 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:10:30.838463 (+ 111us) client_negotiation.cc:769] Sending connection context
0810 04:10:30.838553 (+ 90us) client_negotiation.cc:240] Negotiation successful
0810 04:10:30.838667 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":96,"threads_started":1}
I20260810 04:10:30.839358 19789 heartbeater.cc:347] Connected to a master server at 127.31.32.62:41205
I20260810 04:10:30.839449 19789 heartbeater.cc:464] Registering TS with master...
I20260810 04:10:30.839594 19789 heartbeater.cc:511] Master 127.31.32.62:41205 requested a full tablet report, sending...
I20260810 04:10:30.839967 19290 ts_manager.cc:194] Registered new tserver with Master: afce3cfcf56347aabad5114eb7614802 (127.31.32.3:40159)
I20260810 04:10:30.840370 19290 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.32.3@KRBTEST.COM'} at 127.31.32.3:39345
I20260810 04:10:30.848886 31872 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 04:10:30.856819 19332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:30.850251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53998 (local address 127.31.32.62:41205)
0810 04:10:30.850338 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0810 04:10:30.850340 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:10:30.850448 (+ 108us) server_negotiation.cc:407] Connection header received
0810 04:10:30.850541 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:10:30.850542 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:10:30.850552 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:10:30.850608 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:10:30.851172 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:30.851648 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:30.852479 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:30.852703 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:30.852779 (+ 76us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:10:30.854084 (+ 1305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:10:30.854085 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:10:30.854086 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:10:30.854093 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:10:30.855259 (+ 1166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:10:30.855696 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:10:30.855698 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:10:30.855698 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:10:30.855760 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:10:30.856218 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:10:30.856219 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:10:30.856220 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:10:30.856387 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:10:30.856510 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0810 04:10:30.856655 (+ 145us) server_negotiation.cc:299] Negotiation successful
0810 04:10:30.856720 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260810 04:10:30.858369 19290 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53998:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 04:10:30.859957 19290 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 04:10:30.866562 19805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:30.862299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.3:40159 (local address 127.0.0.1:38132)
0810 04:10:30.862826 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0810 04:10:30.862835 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 04:10:30.862879 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:10:30.863281 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:10:30.863281 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:10:30.863283 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:10:30.863314 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:10:30.863315 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:10:30.864258 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:10:30.864259 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:10:30.865377 (+ 1118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:10:30.865380 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:10:30.866292 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:10:30.866293 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:10:30.866360 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:10:30.866366 (+ 6us) client_negotiation.cc:769] Sending connection context
0810 04:10:30.866405 (+ 39us) client_negotiation.cc:240] Negotiation successful
0810 04:10:30.866463 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"mutex_wait_us":31,"thread_start_us":87,"threads_started":1}
I20260810 04:10:30.866649 19806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:30.862421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57840 (local address 127.31.32.1:40765)
0810 04:10:30.862647 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0810 04:10:30.862652 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 04:10:30.862661 (+ 9us) server_negotiation.cc:407] Connection header received
0810 04:10:30.862697 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:10:30.862699 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:10:30.862721 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:10:30.862808 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:10:30.862973 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:30.864087 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:30.865655 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:30.866317 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:30.866390 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:10:30.866450 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0810 04:10:30.866514 (+ 64us) server_negotiation.cc:299] Negotiation successful
0810 04:10:30.866556 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":85,"threads_started":1}
I20260810 04:10:30.866699 19802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:30.862297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.1:40765 (local address 127.0.0.1:57840)
0810 04:10:30.862459 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0810 04:10:30.862499 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0810 04:10:30.862568 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:10:30.862813 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:10:30.862816 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:10:30.862818 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:10:30.862899 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:10:30.862901 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:10:30.864205 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:10:30.864206 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:10:30.865563 (+ 1357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:10:30.865566 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:10:30.866391 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:10:30.866392 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:10:30.866449 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:10:30.866452 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 04:10:30.866503 (+ 51us) client_negotiation.cc:240] Negotiation successful
0810 04:10:30.866540 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"mutex_wait_us":60,"thread_start_us":59,"threads_started":1}
I20260810 04:10:30.866909 19804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:30.862299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.2:34333 (local address 127.0.0.1:41824)
0810 04:10:30.862931 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0810 04:10:30.862938 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0810 04:10:30.862998 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:10:30.863259 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:10:30.863260 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:10:30.863262 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:10:30.863323 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:10:30.863324 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:10:30.864566 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:10:30.864566 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:10:30.865634 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:10:30.865636 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:10:30.866257 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:10:30.866258 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:10:30.866339 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:10:30.866353 (+ 14us) client_negotiation.cc:769] Sending connection context
0810 04:10:30.866424 (+ 71us) client_negotiation.cc:240] Negotiation successful
0810 04:10:30.866467 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":622,"mutex_wait_us":6,"thread_start_us":107,"threads_started":1}
I20260810 04:10:30.866947 19807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:30.862539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38132 (local address 127.31.32.3:40159)
0810 04:10:30.863123 (+ 584us) server_negotiation.cc:206] Beginning negotiation
0810 04:10:30.863128 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 04:10:30.863137 (+ 9us) server_negotiation.cc:407] Connection header received
0810 04:10:30.863172 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:10:30.863174 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:10:30.863188 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:10:30.863388 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:10:30.863458 (+ 70us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:30.864091 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:30.865468 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:30.866187 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:30.866699 (+ 512us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:10:30.866770 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0810 04:10:30.866795 (+ 25us) server_negotiation.cc:299] Negotiation successful
0810 04:10:30.866845 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":544,"thread_start_us":81,"threads_started":1}
I20260810 04:10:30.867617 19808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:30.862557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41824 (local address 127.31.32.2:34333)
0810 04:10:30.862769 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0810 04:10:30.862773 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 04:10:30.863000 (+ 227us) server_negotiation.cc:407] Connection header received
0810 04:10:30.863073 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:10:30.863076 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:10:30.863091 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:10:30.863168 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:10:30.863389 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:30.864461 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:30.865697 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:30.866166 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:30.866255 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:10:30.866327 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0810 04:10:30.866584 (+ 257us) server_negotiation.cc:299] Negotiation successful
0810 04:10:30.867530 (+ 946us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":79,"threads_started":1}
I20260810 04:10:30.867518 19425 tablet_service.cc:1467] Processing CreateTablet for tablet 5d14469c9b454d439766b5802b218042 (DEFAULT_TABLE table=test-table [id=238a55ffc8c04f3d8405437fbf78f1b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:10:30.867771 19719 tablet_service.cc:1467] Processing CreateTablet for tablet 5d14469c9b454d439766b5802b218042 (DEFAULT_TABLE table=test-table [id=238a55ffc8c04f3d8405437fbf78f1b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:10:30.867923 19425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d14469c9b454d439766b5802b218042. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:10:30.868112 19719 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d14469c9b454d439766b5802b218042. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:10:30.868564 19574 tablet_service.cc:1467] Processing CreateTablet for tablet 5d14469c9b454d439766b5802b218042 (DEFAULT_TABLE table=test-table [id=238a55ffc8c04f3d8405437fbf78f1b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:10:30.868894 19574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d14469c9b454d439766b5802b218042. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:10:30.870895 19809 tablet_bootstrap.cc:492] T 5d14469c9b454d439766b5802b218042 P 7dbbd9270f9e433d8fd19411edf43d81: Bootstrap starting.
I20260810 04:10:30.871513 19811 tablet_bootstrap.cc:492] T 5d14469c9b454d439766b5802b218042 P afce3cfcf56347aabad5114eb7614802: Bootstrap starting.
I20260810 04:10:30.871644 19809 tablet_bootstrap.cc:654] T 5d14469c9b454d439766b5802b218042 P 7dbbd9270f9e433d8fd19411edf43d81: Neither blocks nor log segments found. Creating new log.
I20260810 04:10:30.872164 19812 tablet_bootstrap.cc:492] T 5d14469c9b454d439766b5802b218042 P b09b28d0d79a4711ad09c8c43786741b: Bootstrap starting.
I20260810 04:10:30.872254 19811 tablet_bootstrap.cc:654] T 5d14469c9b454d439766b5802b218042 P afce3cfcf56347aabad5114eb7614802: Neither blocks nor log segments found. Creating new log.
I20260810 04:10:30.872438 19809 log.cc:824] T 5d14469c9b454d439766b5802b218042 P 7dbbd9270f9e433d8fd19411edf43d81: Log is configured to *not* fsync() on all Append() calls
I20260810 04:10:30.872880 19812 tablet_bootstrap.cc:654] T 5d14469c9b454d439766b5802b218042 P b09b28d0d79a4711ad09c8c43786741b: Neither blocks nor log segments found. Creating new log.
I20260810 04:10:30.872884 19811 log.cc:824] T 5d14469c9b454d439766b5802b218042 P afce3cfcf56347aabad5114eb7614802: Log is configured to *not* fsync() on all Append() calls
I20260810 04:10:30.873307 19809 tablet_bootstrap.cc:492] T 5d14469c9b454d439766b5802b218042 P 7dbbd9270f9e433d8fd19411edf43d81: No bootstrap required, opened a new log
I20260810 04:10:30.873384 19809 ts_tablet_manager.cc:1397] T 5d14469c9b454d439766b5802b218042 P 7dbbd9270f9e433d8fd19411edf43d81: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 04:10:30.873489 19812 log.cc:824] T 5d14469c9b454d439766b5802b218042 P b09b28d0d79a4711ad09c8c43786741b: Log is configured to *not* fsync() on all Append() calls
I20260810 04:10:30.873674 19811 tablet_bootstrap.cc:492] T 5d14469c9b454d439766b5802b218042 P afce3cfcf56347aabad5114eb7614802: No bootstrap required, opened a new log
I20260810 04:10:30.873739 19811 ts_tablet_manager.cc:1397] T 5d14469c9b454d439766b5802b218042 P afce3cfcf56347aabad5114eb7614802: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260810 04:10:30.874184 19812 tablet_bootstrap.cc:492] T 5d14469c9b454d439766b5802b218042 P b09b28d0d79a4711ad09c8c43786741b: No bootstrap required, opened a new log
I20260810 04:10:30.874257 19812 ts_tablet_manager.cc:1397] T 5d14469c9b454d439766b5802b218042 P b09b28d0d79a4711ad09c8c43786741b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 04:10:30.875165 19811 raft_consensus.cc:348] T 5d14469c9b454d439766b5802b218042 P afce3cfcf56347aabad5114eb7614802 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09b28d0d79a4711ad09c8c43786741b" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 34333 } } peers { permanent_uuid: "7dbbd9270f9e433d8fd19411edf43d81" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 40765 } } peers { permanent_uuid: "afce3cfcf56347aabad5114eb7614802" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 40159 } }
I20260810 04:10:30.875341 19811 raft_consensus.cc:374] T 5d14469c9b454d439766b5802b218042 P afce3cfcf56347aabad5114eb7614802 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:10:30.875389 19811 raft_consensus.cc:729] T 5d14469c9b454d439766b5802b218042 P afce3cfcf56347aabad5114eb7614802 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: afce3cfcf56347aabad5114eb7614802, State: Initialized, Role: FOLLOWER
I20260810 04:10:30.875519 19811 consensus_queue.cc:260] T 5d14469c9b454d439766b5802b218042 P afce3cfcf56347aabad5114eb7614802 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09b28d0d79a4711ad09c8c43786741b" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 34333 } } peers { permanent_uuid: "7dbbd9270f9e433d8fd19411edf43d81" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 40765 } } peers { permanent_uuid: "afce3cfcf56347aabad5114eb7614802" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 40159 } }
I20260810 04:10:30.875478 19809 raft_consensus.cc:348] T 5d14469c9b454d439766b5802b218042 P 7dbbd9270f9e433d8fd19411edf43d81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09b28d0d79a4711ad09c8c43786741b" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 34333 } } peers { permanent_uuid: "7dbbd9270f9e433d8fd19411edf43d81" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 40765 } } peers { permanent_uuid: "afce3cfcf56347aabad5114eb7614802" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 40159 } }
I20260810 04:10:30.875658 19809 raft_consensus.cc:374] T 5d14469c9b454d439766b5802b218042 P 7dbbd9270f9e433d8fd19411edf43d81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:10:30.875692 19809 raft_consensus.cc:729] T 5d14469c9b454d439766b5802b218042 P 7dbbd9270f9e433d8fd19411edf43d81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7dbbd9270f9e433d8fd19411edf43d81, State: Initialized, Role: FOLLOWER
I20260810 04:10:30.875809 19811 ts_tablet_manager.cc:1428] T 5d14469c9b454d439766b5802b218042 P afce3cfcf56347aabad5114eb7614802: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260810 04:10:30.875798 19809 consensus_queue.cc:260] T 5d14469c9b454d439766b5802b218042 P 7dbbd9270f9e433d8fd19411edf43d81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09b28d0d79a4711ad09c8c43786741b" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 34333 } } peers { permanent_uuid: "7dbbd9270f9e433d8fd19411edf43d81" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 40765 } } peers { permanent_uuid: "afce3cfcf56347aabad5114eb7614802" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 40159 } }
I20260810 04:10:30.875998 19789 heartbeater.cc:503] Master 127.31.32.62:41205 was elected leader, sending a full tablet report...
I20260810 04:10:30.876111 19809 ts_tablet_manager.cc:1428] T 5d14469c9b454d439766b5802b218042 P 7dbbd9270f9e433d8fd19411edf43d81: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 04:10:30.876261 19499 heartbeater.cc:503] Master 127.31.32.62:41205 was elected leader, sending a full tablet report...
I20260810 04:10:30.876274 19812 raft_consensus.cc:348] T 5d14469c9b454d439766b5802b218042 P b09b28d0d79a4711ad09c8c43786741b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09b28d0d79a4711ad09c8c43786741b" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 34333 } } peers { permanent_uuid: "7dbbd9270f9e433d8fd19411edf43d81" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 40765 } } peers { permanent_uuid: "afce3cfcf56347aabad5114eb7614802" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 40159 } }
I20260810 04:10:30.876439 19812 raft_consensus.cc:374] T 5d14469c9b454d439766b5802b218042 P b09b28d0d79a4711ad09c8c43786741b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:10:30.876479 19812 raft_consensus.cc:729] T 5d14469c9b454d439766b5802b218042 P b09b28d0d79a4711ad09c8c43786741b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b09b28d0d79a4711ad09c8c43786741b, State: Initialized, Role: FOLLOWER
I20260810 04:10:30.876590 19812 consensus_queue.cc:260] T 5d14469c9b454d439766b5802b218042 P b09b28d0d79a4711ad09c8c43786741b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09b28d0d79a4711ad09c8c43786741b" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 34333 } } peers { permanent_uuid: "7dbbd9270f9e433d8fd19411edf43d81" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 40765 } } peers { permanent_uuid: "afce3cfcf56347aabad5114eb7614802" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 40159 } }
I20260810 04:10:30.876870 19812 ts_tablet_manager.cc:1428] T 5d14469c9b454d439766b5802b218042 P b09b28d0d79a4711ad09c8c43786741b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 04:10:30.876907 19643 heartbeater.cc:503] Master 127.31.32.62:41205 was elected leader, sending a full tablet report...
W20260810 04:10:30.906438 19644 tablet.cc:2369] T 5d14469c9b454d439766b5802b218042 P b09b28d0d79a4711ad09c8c43786741b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 04:10:30.936973 19817 raft_consensus.cc:482] T 5d14469c9b454d439766b5802b218042 P afce3cfcf56347aabad5114eb7614802 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 04:10:30.937116 19817 raft_consensus.cc:504] T 5d14469c9b454d439766b5802b218042 P afce3cfcf56347aabad5114eb7614802 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09b28d0d79a4711ad09c8c43786741b" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 34333 } } peers { permanent_uuid: "7dbbd9270f9e433d8fd19411edf43d81" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 40765 } } peers { permanent_uuid: "afce3cfcf56347aabad5114eb7614802" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 40159 } }
I20260810 04:10:30.937494 19817 leader_election.cc:288] T 5d14469c9b454d439766b5802b218042 P afce3cfcf56347aabad5114eb7614802 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b09b28d0d79a4711ad09c8c43786741b (127.31.32.2:34333), 7dbbd9270f9e433d8fd19411edf43d81 (127.31.32.1:40765)
I20260810 04:10:30.940541 19808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:30.937762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.3:37701 (local address 127.31.32.2:34333)
0810 04:10:30.937857 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0810 04:10:30.937859 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:10:30.937988 (+ 129us) server_negotiation.cc:407] Connection header received
0810 04:10:30.938053 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:10:30.938054 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:10:30.938067 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:10:30.938142 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:10:30.938300 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:30.938759 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:30.939694 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:30.940083 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:30.940250 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:10:30.940263 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0810 04:10:30.940432 (+ 169us) server_negotiation.cc:299] Negotiation successful
0810 04:10:30.940473 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260810 04:10:30.940541 19806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:30.937789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.3:55337 (local address 127.31.32.1:40765)
0810 04:10:30.937847 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0810 04:10:30.937850 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:10:30.937864 (+ 14us) server_negotiation.cc:407] Connection header received
0810 04:10:30.937899 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:10:30.937900 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:10:30.937913 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:10:30.938010 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:10:30.938194 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:30.938683 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:30.939665 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:30.940106 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:30.940250 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:10:30.940271 (+ 21us) server_negotiation.cc:1035] Waiting for connection context
0810 04:10:30.940425 (+ 154us) server_negotiation.cc:299] Negotiation successful
0810 04:10:30.940467 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260810 04:10:30.940547 19791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:30.937690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.1:40765 (local address 127.31.32.3:55337)
0810 04:10:30.937750 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0810 04:10:30.937763 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 04:10:30.937807 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:10:30.938032 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:10:30.938034 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:10:30.938036 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:10:30.938112 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:10:30.938114 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:10:30.938781 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:10:30.938782 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:10:30.939562 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:10:30.939564 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:10:30.940254 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:10:30.940255 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:10:30.940344 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:10:30.940347 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 04:10:30.940419 (+ 72us) client_negotiation.cc:240] Negotiation successful
0810 04:10:30.940464 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260810 04:10:30.940558 19820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:30.937704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.2:34333 (local address 127.31.32.3:37701)
0810 04:10:30.937926 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0810 04:10:30.937934 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0810 04:10:30.937997 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:10:30.938147 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:10:30.938148 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:10:30.938150 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:10:30.938229 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:10:30.938230 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:10:30.938857 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:10:30.938858 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:10:30.939599 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:10:30.939601 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:10:30.940256 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:10:30.940257 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:10:30.940342 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:10:30.940346 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 04:10:30.940413 (+ 67us) client_negotiation.cc:240] Negotiation successful
0810 04:10:30.940456 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":146,"threads_started":1}
I20260810 04:10:30.940981 19594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d14469c9b454d439766b5802b218042" candidate_uuid: "afce3cfcf56347aabad5114eb7614802" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b09b28d0d79a4711ad09c8c43786741b" is_pre_election: true
I20260810 04:10:30.941056 19447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d14469c9b454d439766b5802b218042" candidate_uuid: "afce3cfcf56347aabad5114eb7614802" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7dbbd9270f9e433d8fd19411edf43d81" is_pre_election: true
I20260810 04:10:30.941166 19594 raft_consensus.cc:2454] T 5d14469c9b454d439766b5802b218042 P b09b28d0d79a4711ad09c8c43786741b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate afce3cfcf56347aabad5114eb7614802 in term 0.
I20260810 04:10:30.941239 19447 raft_consensus.cc:2454] T 5d14469c9b454d439766b5802b218042 P 7dbbd9270f9e433d8fd19411edf43d81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate afce3cfcf56347aabad5114eb7614802 in term 0.
I20260810 04:10:30.941437 19666 leader_election.cc:302] T 5d14469c9b454d439766b5802b218042 P afce3cfcf56347aabad5114eb7614802 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: afce3cfcf56347aabad5114eb7614802, b09b28d0d79a4711ad09c8c43786741b; no voters:
I20260810 04:10:30.941599 19817 raft_consensus.cc:2781] T 5d14469c9b454d439766b5802b218042 P afce3cfcf56347aabad5114eb7614802 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 04:10:30.941660 19817 raft_consensus.cc:482] T 5d14469c9b454d439766b5802b218042 P afce3cfcf56347aabad5114eb7614802 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 04:10:30.941684 19817 raft_consensus.cc:3037] T 5d14469c9b454d439766b5802b218042 P afce3cfcf56347aabad5114eb7614802 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:10:30.942408 19817 raft_consensus.cc:504] T 5d14469c9b454d439766b5802b218042 P afce3cfcf56347aabad5114eb7614802 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09b28d0d79a4711ad09c8c43786741b" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 34333 } } peers { permanent_uuid: "7dbbd9270f9e433d8fd19411edf43d81" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 40765 } } peers { permanent_uuid: "afce3cfcf56347aabad5114eb7614802" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 40159 } }
I20260810 04:10:30.942559 19817 leader_election.cc:288] T 5d14469c9b454d439766b5802b218042 P afce3cfcf56347aabad5114eb7614802 [CANDIDATE]: Term 1 election: Requested vote from peers b09b28d0d79a4711ad09c8c43786741b (127.31.32.2:34333), 7dbbd9270f9e433d8fd19411edf43d81 (127.31.32.1:40765)
I20260810 04:10:30.942729 19594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d14469c9b454d439766b5802b218042" candidate_uuid: "afce3cfcf56347aabad5114eb7614802" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b09b28d0d79a4711ad09c8c43786741b"
I20260810 04:10:30.942750 19447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d14469c9b454d439766b5802b218042" candidate_uuid: "afce3cfcf56347aabad5114eb7614802" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7dbbd9270f9e433d8fd19411edf43d81"
I20260810 04:10:30.942814 19594 raft_consensus.cc:3037] T 5d14469c9b454d439766b5802b218042 P b09b28d0d79a4711ad09c8c43786741b [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:10:30.942864 19447 raft_consensus.cc:3037] T 5d14469c9b454d439766b5802b218042 P 7dbbd9270f9e433d8fd19411edf43d81 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:10:30.943522 19594 raft_consensus.cc:2454] T 5d14469c9b454d439766b5802b218042 P b09b28d0d79a4711ad09c8c43786741b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate afce3cfcf56347aabad5114eb7614802 in term 1.
I20260810 04:10:30.943686 19666 leader_election.cc:302] T 5d14469c9b454d439766b5802b218042 P afce3cfcf56347aabad5114eb7614802 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: afce3cfcf56347aabad5114eb7614802, b09b28d0d79a4711ad09c8c43786741b; no voters:
I20260810 04:10:30.943720 19447 raft_consensus.cc:2454] T 5d14469c9b454d439766b5802b218042 P 7dbbd9270f9e433d8fd19411edf43d81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate afce3cfcf56347aabad5114eb7614802 in term 1.
I20260810 04:10:30.943769 19817 raft_consensus.cc:2781] T 5d14469c9b454d439766b5802b218042 P afce3cfcf56347aabad5114eb7614802 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:10:30.944013 19817 raft_consensus.cc:686] T 5d14469c9b454d439766b5802b218042 P afce3cfcf56347aabad5114eb7614802 [term 1 LEADER]: Becoming Leader. State: Replica: afce3cfcf56347aabad5114eb7614802, State: Running, Role: LEADER
I20260810 04:10:30.944159 19817 consensus_queue.cc:237] T 5d14469c9b454d439766b5802b218042 P afce3cfcf56347aabad5114eb7614802 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09b28d0d79a4711ad09c8c43786741b" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 34333 } } peers { permanent_uuid: "7dbbd9270f9e433d8fd19411edf43d81" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 40765 } } peers { permanent_uuid: "afce3cfcf56347aabad5114eb7614802" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 40159 } }
I20260810 04:10:30.945133 19290 catalog_manager.cc:5495] T 5d14469c9b454d439766b5802b218042 P afce3cfcf56347aabad5114eb7614802 reported cstate change: term changed from 0 to 1, leader changed from <none> to afce3cfcf56347aabad5114eb7614802 (127.31.32.3). New cstate: current_term: 1 leader_uuid: "afce3cfcf56347aabad5114eb7614802" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09b28d0d79a4711ad09c8c43786741b" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 34333 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7dbbd9270f9e433d8fd19411edf43d81" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 40765 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "afce3cfcf56347aabad5114eb7614802" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 40159 } health_report { overall_health: HEALTHY } } }
I20260810 04:10:30.958050 19807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:30.955814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38142 (local address 127.31.32.3:40159)
0810 04:10:30.955871 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0810 04:10:30.955873 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:10:30.955880 (+ 7us) server_negotiation.cc:407] Connection header received
0810 04:10:30.955911 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:10:30.955912 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:10:30.955930 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:10:30.956005 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 04:10:30.956176 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:30.956672 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:30.957391 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:30.957570 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:30.957625 (+ 55us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:10:30.957770 (+ 145us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 04:10:30.957854 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 04:10:30.957927 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0810 04:10:30.957973 (+ 46us) server_negotiation.cc:299] Negotiation successful
0810 04:10:30.957980 (+ 7us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260810 04:10:30.959887 19447 raft_consensus.cc:1261] T 5d14469c9b454d439766b5802b218042 P 7dbbd9270f9e433d8fd19411edf43d81 [term 1 FOLLOWER]: Refusing update from remote peer afce3cfcf56347aabad5114eb7614802: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:10:30.959892 19594 raft_consensus.cc:1261] T 5d14469c9b454d439766b5802b218042 P b09b28d0d79a4711ad09c8c43786741b [term 1 FOLLOWER]: Refusing update from remote peer afce3cfcf56347aabad5114eb7614802: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:10:30.960279 19821 consensus_queue.cc:1035] T 5d14469c9b454d439766b5802b218042 P afce3cfcf56347aabad5114eb7614802 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b09b28d0d79a4711ad09c8c43786741b" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 34333 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:10:30.960418 19817 consensus_queue.cc:1035] T 5d14469c9b454d439766b5802b218042 P afce3cfcf56347aabad5114eb7614802 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7dbbd9270f9e433d8fd19411edf43d81" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 40765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:10:30.962828 19825 mvcc.cc:205] Tried to move back new op lower bound from 7316828286809001984 to 7316828286748221440. Current Snapshot: MvccSnapshot[applied={T|T < 7316828286809001984}]
I20260810 04:10:30.965742 19290 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53998:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 04:10:30.965838 19290 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53998:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 04:10:30.966549 19290 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5c052a9a931d4736b3f86a80ee809941: Sending DeleteTablet for 3 replicas of tablet 5d14469c9b454d439766b5802b218042
I20260810 04:10:30.966856 19574 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d14469c9b454d439766b5802b218042 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:10:30 UTC) from {username='kudu', principal='kudu/127.31.32.62@KRBTEST.COM'} at 127.0.0.1:41824
I20260810 04:10:30.966955 19425 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d14469c9b454d439766b5802b218042 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:10:30 UTC) from {username='kudu', principal='kudu/127.31.32.62@KRBTEST.COM'} at 127.0.0.1:57840
I20260810 04:10:30.967137 19834 tablet_replica.cc:330] T 5d14469c9b454d439766b5802b218042 P b09b28d0d79a4711ad09c8c43786741b: stopping tablet replica
I20260810 04:10:30.967250 19834 raft_consensus.cc:2229] T 5d14469c9b454d439766b5802b218042 P b09b28d0d79a4711ad09c8c43786741b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 04:10:30.967285 31872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskws89Io/build/release/bin/kudu with pid 19358
I20260810 04:10:30.967365 19834 raft_consensus.cc:2258] T 5d14469c9b454d439766b5802b218042 P b09b28d0d79a4711ad09c8c43786741b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:10:30.967401 19719 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d14469c9b454d439766b5802b218042 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:10:30 UTC) from {username='kudu', principal='kudu/127.31.32.62@KRBTEST.COM'} at 127.0.0.1:38132
I20260810 04:10:30.969023 19836 tablet_replica.cc:330] T 5d14469c9b454d439766b5802b218042 P afce3cfcf56347aabad5114eb7614802: stopping tablet replica
I20260810 04:10:30.969211 19836 raft_consensus.cc:2229] T 5d14469c9b454d439766b5802b218042 P afce3cfcf56347aabad5114eb7614802 [term 1 LEADER]: Raft consensus shutting down.
I20260810 04:10:30.969409 19836 raft_consensus.cc:2258] T 5d14469c9b454d439766b5802b218042 P afce3cfcf56347aabad5114eb7614802 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:10:30.970465 19836 ts_tablet_manager.cc:1905] T 5d14469c9b454d439766b5802b218042 P afce3cfcf56347aabad5114eb7614802: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:10:30.972745 19836 ts_tablet_manager.cc:1918] T 5d14469c9b454d439766b5802b218042 P afce3cfcf56347aabad5114eb7614802: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 04:10:30.972967 19836 log.cc:1196] T 5d14469c9b454d439766b5802b218042 P afce3cfcf56347aabad5114eb7614802: Deleting WAL directory at /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786334794676589-31872-0/minicluster-data/ts-2/wal/wals/5d14469c9b454d439766b5802b218042
I20260810 04:10:30.973477 19836 ts_tablet_manager.cc:1939] T 5d14469c9b454d439766b5802b218042 P afce3cfcf56347aabad5114eb7614802: Deleting consensus metadata
I20260810 04:10:30.973932 19272 catalog_manager.cc:4841] TS afce3cfcf56347aabad5114eb7614802 (127.31.32.3:40159): tablet 5d14469c9b454d439766b5802b218042 (table test-table [id=238a55ffc8c04f3d8405437fbf78f1b1]) successfully deleted
I20260810 04:10:30.974121 19834 ts_tablet_manager.cc:1905] T 5d14469c9b454d439766b5802b218042 P b09b28d0d79a4711ad09c8c43786741b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:10:30.976392 19834 ts_tablet_manager.cc:1918] T 5d14469c9b454d439766b5802b218042 P b09b28d0d79a4711ad09c8c43786741b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 04:10:30.976475 19834 log.cc:1196] T 5d14469c9b454d439766b5802b218042 P b09b28d0d79a4711ad09c8c43786741b: Deleting WAL directory at /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786334794676589-31872-0/minicluster-data/ts-1/wal/wals/5d14469c9b454d439766b5802b218042
I20260810 04:10:30.976763 19834 ts_tablet_manager.cc:1939] T 5d14469c9b454d439766b5802b218042 P b09b28d0d79a4711ad09c8c43786741b: Deleting consensus metadata
I20260810 04:10:30.977247 19273 catalog_manager.cc:4841] TS b09b28d0d79a4711ad09c8c43786741b (127.31.32.2:34333): tablet 5d14469c9b454d439766b5802b218042 (table test-table [id=238a55ffc8c04f3d8405437fbf78f1b1]) successfully deleted
W20260810 04:10:30.978786 19274 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.32.1:40765 (error 108)
I20260810 04:10:30.979103 31872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskws89Io/build/release/bin/kudu with pid 19504
I20260810 04:10:30.980185 19802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:30.979224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.1:40765 (local address 127.0.0.1:57842)
0810 04:10:30.980055 (+ 831us) negotiation.cc:107] Waiting for socket to connect
0810 04:10:30.980108 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.1:40765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":805}
W20260810 04:10:30.980644 19274 catalog_manager.cc:4584] TS 7dbbd9270f9e433d8fd19411edf43d81 (127.31.32.1:40765): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5d14469c9b454d439766b5802b218042: Network error: Client connection negotiation failed: client connection to 127.31.32.1:40765: connect: Connection refused (error 111)
I20260810 04:10:30.986233 31872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskws89Io/build/release/bin/kudu with pid 19648
I20260810 04:10:30.992523 31872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskws89Io/build/release/bin/kudu with pid 19253
2026-08-10T04:10:30Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2055 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-08-10T04:10:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T04:10:31Z Disabled control of system clock
I20260810 04:10:31.023160 31872 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskws89Io/build/release/bin/kudu
/tmp/dist-test-taskws89Io/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786334794676589-31872-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786334794676589-31872-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786334794676589-31872-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786334794676589-31872-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.32.62:38155
--webserver_interface=127.31.32.62
--webserver_port=0
--builtin_ntp_servers=127.31.32.20:40509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.32.62:38155
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 04:10:31.106719 19841 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:10:31.106945 19841 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:10:31.106966 19841 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:10:31.108155 19841 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:10:31.108194 19841 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:10:31.108206 19841 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:10:31.108219 19841 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:10:31.108232 19841 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:10:31.109501 19841 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.32.20:40509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786334794676589-31872-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786334794676589-31872-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.32.62:38155
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.32.62:38155
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786334794676589-31872-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.32.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19841
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786334794676589-31872-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:10:31.109694 19841 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:10:31.109925 19841 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:10:31.115131 19851 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:10:31.115242 19848 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:10:31.115145 19849 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:10:31.115437 19841 server_base.cc:1034] running on GCE node
I20260810 04:10:31.115732 19841 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:10:31.115983 19841 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:10:31.117174 19841 hybrid_clock.cc:648] HybridClock initialized: now 1786335031117162 us; error 33 us; skew 500 ppm
I20260810 04:10:31.118618 19841 webserver.cc:460] Webserver started at http://127.31.32.62:40933/ using document root <none> and password file <none>
I20260810 04:10:31.118870 19841 fs_manager.cc:359] Metadata directory not provided
I20260810 04:10:31.118920 19841 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:10:31.119097 19841 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:10:31.120239 19841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786334794676589-31872-0/minicluster-data/master-0/data/instance:
uuid: "80339afd93d945e08e147eff38c7fe7c"
format_stamp: "Formatted at 2026-08-10 04:10:31 on dist-test-slave-8w9v"
server_key: "1b5c174aa3d2d694098f9c6b0501ea5f"
server_key_iv: "e6ad003f7bba35dcce163f775b1b7c6b"
server_key_version: "encryptionkey@0"
I20260810 04:10:31.120615 19841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786334794676589-31872-0/minicluster-data/master-0/wal/instance:
uuid: "80339afd93d945e08e147eff38c7fe7c"
format_stamp: "Formatted at 2026-08-10 04:10:31 on dist-test-slave-8w9v"
server_key: "1b5c174aa3d2d694098f9c6b0501ea5f"
server_key_iv: "e6ad003f7bba35dcce163f775b1b7c6b"
server_key_version: "encryptionkey@0"
I20260810 04:10:31.121979 19841 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 04:10:31.122848 19858 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:10:31.123059 19841 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:10:31.123162 19841 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786334794676589-31872-0/minicluster-data/master-0/data,/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786334794676589-31872-0/minicluster-data/master-0/wal
uuid: "80339afd93d945e08e147eff38c7fe7c"
format_stamp: "Formatted at 2026-08-10 04:10:31 on dist-test-slave-8w9v"
server_key: "1b5c174aa3d2d694098f9c6b0501ea5f"
server_key_iv: "e6ad003f7bba35dcce163f775b1b7c6b"
server_key_version: "encryptionkey@0"
I20260810 04:10:31.123282 19841 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786334794676589-31872-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786334794676589-31872-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786334794676589-31872-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:10:31.150743 19841 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:10:31.151129 19841 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:10:31.151328 19841 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:10:31.155674 19841 rpc_server.cc:307] RPC server started. Bound to: 127.31.32.62:38155
I20260810 04:10:31.155740 19916 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.32.62:38155 every 8 connection(s)
I20260810 04:10:31.156145 19841 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786334794676589-31872-0/minicluster-data/master-0/data/info.pb
I20260810 04:10:31.157189 19917 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:10:31.158233 31872 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskws89Io/build/release/bin/kudu as pid 19841
I20260810 04:10:31.158322 31872 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786334794676589-31872-0/minicluster-data/master-0/wal/instance
I20260810 04:10:31.158543 31872 external_mini_cluster.cc:1428] Setting key 31763d6089f8fcbe23a5b6412f2bc075
I20260810 04:10:31.160435 19917 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 80339afd93d945e08e147eff38c7fe7c: Bootstrap starting.
I20260810 04:10:31.161183 19917 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 80339afd93d945e08e147eff38c7fe7c: Neither blocks nor log segments found. Creating new log.
I20260810 04:10:31.161504 19917 log.cc:824] T 00000000000000000000000000000000 P 80339afd93d945e08e147eff38c7fe7c: Log is configured to *not* fsync() on all Append() calls
I20260810 04:10:31.162256 19917 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 80339afd93d945e08e147eff38c7fe7c: No bootstrap required, opened a new log
I20260810 04:10:31.164657 19917 raft_consensus.cc:348] T 00000000000000000000000000000000 P 80339afd93d945e08e147eff38c7fe7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80339afd93d945e08e147eff38c7fe7c" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 38155 } }
I20260810 04:10:31.164860 19917 raft_consensus.cc:374] T 00000000000000000000000000000000 P 80339afd93d945e08e147eff38c7fe7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:10:31.164924 19917 raft_consensus.cc:729] T 00000000000000000000000000000000 P 80339afd93d945e08e147eff38c7fe7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80339afd93d945e08e147eff38c7fe7c, State: Initialized, Role: FOLLOWER
I20260810 04:10:31.165076 19920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:31.159120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51260 (local address 127.31.32.62:38155)
0810 04:10:31.159356 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0810 04:10:31.159366 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 04:10:31.159401 (+ 35us) server_negotiation.cc:407] Connection header received
0810 04:10:31.160094 (+ 693us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:10:31.160104 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:10:31.160216 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:10:31.160382 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:10:31.160629 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:31.161947 (+ 1318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:31.162771 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:31.163404 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:31.163532 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:10:31.164291 (+ 759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:10:31.164297 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:10:31.164297 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:10:31.164519 (+ 222us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:10:31.164564 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:10:31.164578 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:10:31.164653 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0810 04:10:31.164716 (+ 63us) server_negotiation.cc:299] Negotiation successful
0810 04:10:31.164796 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":83,"threads_started":1}
I20260810 04:10:31.165066 19917 consensus_queue.cc:260] T 00000000000000000000000000000000 P 80339afd93d945e08e147eff38c7fe7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80339afd93d945e08e147eff38c7fe7c" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 38155 } }
I20260810 04:10:31.165158 19917 raft_consensus.cc:388] T 00000000000000000000000000000000 P 80339afd93d945e08e147eff38c7fe7c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 04:10:31.165190 19917 raft_consensus.cc:482] T 00000000000000000000000000000000 P 80339afd93d945e08e147eff38c7fe7c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 04:10:31.165233 19917 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 80339afd93d945e08e147eff38c7fe7c [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:10:31.166000 19917 raft_consensus.cc:504] T 00000000000000000000000000000000 P 80339afd93d945e08e147eff38c7fe7c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80339afd93d945e08e147eff38c7fe7c" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 38155 } }
I20260810 04:10:31.166121 19917 leader_election.cc:302] T 00000000000000000000000000000000 P 80339afd93d945e08e147eff38c7fe7c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 80339afd93d945e08e147eff38c7fe7c; no voters:
I20260810 04:10:31.166307 19917 leader_election.cc:288] T 00000000000000000000000000000000 P 80339afd93d945e08e147eff38c7fe7c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 04:10:31.166363 19922 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 80339afd93d945e08e147eff38c7fe7c [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:10:31.166556 19922 raft_consensus.cc:686] T 00000000000000000000000000000000 P 80339afd93d945e08e147eff38c7fe7c [term 1 LEADER]: Becoming Leader. State: Replica: 80339afd93d945e08e147eff38c7fe7c, State: Running, Role: LEADER
I20260810 04:10:31.166587 19917 sys_catalog.cc:564] T 00000000000000000000000000000000 P 80339afd93d945e08e147eff38c7fe7c [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:10:31.166649 19922 consensus_queue.cc:237] T 00000000000000000000000000000000 P 80339afd93d945e08e147eff38c7fe7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80339afd93d945e08e147eff38c7fe7c" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 38155 } }
I20260810 04:10:31.167097 19923 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80339afd93d945e08e147eff38c7fe7c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "80339afd93d945e08e147eff38c7fe7c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80339afd93d945e08e147eff38c7fe7c" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 38155 } } }
I20260810 04:10:31.167109 19924 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80339afd93d945e08e147eff38c7fe7c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 80339afd93d945e08e147eff38c7fe7c. Latest consensus state: current_term: 1 leader_uuid: "80339afd93d945e08e147eff38c7fe7c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80339afd93d945e08e147eff38c7fe7c" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 38155 } } }
I20260810 04:10:31.167316 19924 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80339afd93d945e08e147eff38c7fe7c [sys.catalog]: This master's current role is: LEADER
I20260810 04:10:31.167533 19923 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80339afd93d945e08e147eff38c7fe7c [sys.catalog]: This master's current role is: LEADER
I20260810 04:10:31.167719 19932 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 04:10:31.168244 19932 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 04:10:31.169756 19932 catalog_manager.cc:1347] Generated new cluster ID: e2939c9e503947f5a08fed039de07da9
I20260810 04:10:31.169808 19932 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 04:10:31.177559 19932 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 04:10:31.178066 19932 catalog_manager.cc:1504] Loading token signing keys...
I20260810 04:10:31.185705 19932 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 80339afd93d945e08e147eff38c7fe7c: Generated new TSK 0
I20260810 04:10:31.185881 19932 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 04:10:31.192839 31872 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskws89Io/build/release/bin/kudu
/tmp/dist-test-taskws89Io/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786334794676589-31872-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786334794676589-31872-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786334794676589-31872-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786334794676589-31872-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.32.1:0
--local_ip_for_outbound_sockets=127.31.32.1
--webserver_interface=127.31.32.1
--webserver_port=0
--tserver_master_addrs=127.31.32.62:38155
--builtin_ntp_servers=127.31.32.20:40509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 04:10:31.277819 19942 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:10:31.278035 19942 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:10:31.278096 19942 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:10:31.279261 19942 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:10:31.279313 19942 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:10:31.279340 19942 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.32.1
I20260810 04:10:31.280836 19942 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.32.20:40509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786334794676589-31872-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786334794676589-31872-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.32.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786334794676589-31872-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.32.1
--webserver_port=0
--tserver_master_addrs=127.31.32.62:38155
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19942
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.32.1
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786334794676589-31872-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:10:31.281095 19942 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:10:31.281445 19942 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:10:31.287730 19949 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:10:31.287730 19952 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:10:31.287725 19947 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:10:31.287951 19942 server_base.cc:1034] running on GCE node
I20260810 04:10:31.288493 19942 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:10:31.288769 19942 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:10:31.289968 19942 hybrid_clock.cc:648] HybridClock initialized: now 1786335031289940 us; error 44 us; skew 500 ppm
I20260810 04:10:31.291450 19942 webserver.cc:460] Webserver started at http://127.31.32.1:42361/ using document root <none> and password file <none>
I20260810 04:10:31.291725 19942 fs_manager.cc:359] Metadata directory not provided
I20260810 04:10:31.291803 19942 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:10:31.291972 19942 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:10:31.293150 19942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786334794676589-31872-0/minicluster-data/ts-0/data/instance:
uuid: "00cad55fe33f45f38e32199b882073a7"
format_stamp: "Formatted at 2026-08-10 04:10:31 on dist-test-slave-8w9v"
server_key: "4288fa0cae360dd8146ba015780ee34f"
server_key_iv: "0a81472838cf6b59d93f1d38c469ad8a"
server_key_version: "encryptionkey@0"
I20260810 04:10:31.293546 19942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786334794676589-31872-0/minicluster-data/ts-0/wal/instance:
uuid: "00cad55fe33f45f38e32199b882073a7"
format_stamp: "Formatted at 2026-08-10 04:10:31 on dist-test-slave-8w9v"
server_key: "4288fa0cae360dd8146ba015780ee34f"
server_key_iv: "0a81472838cf6b59d93f1d38c469ad8a"
server_key_version: "encryptionkey@0"
I20260810 04:10:31.294920 19942 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260810 04:10:31.295683 19959 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:10:31.295855 19942 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 04:10:31.295955 19942 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786334794676589-31872-0/minicluster-data/ts-0/data,/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786334794676589-31872-0/minicluster-data/ts-0/wal
uuid: "00cad55fe33f45f38e32199b882073a7"
format_stamp: "Formatted at 2026-08-10 04:10:31 on dist-test-slave-8w9v"
server_key: "4288fa0cae360dd8146ba015780ee34f"
server_key_iv: "0a81472838cf6b59d93f1d38c469ad8a"
server_key_version: "encryptionkey@0"
I20260810 04:10:31.296063 19942 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786334794676589-31872-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786334794676589-31872-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786334794676589-31872-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:10:31.310505 19942 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:10:31.310834 19942 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:10:31.311005 19942 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:10:31.311259 19942 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:10:31.311609 19942 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:10:31.311671 19942 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:10:31.311712 19942 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:10:31.311740 19942 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:10:31.317945 19942 rpc_server.cc:307] RPC server started. Bound to: 127.31.32.1:33103
I20260810 04:10:31.318011 20086 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.32.1:33103 every 8 connection(s)
I20260810 04:10:31.318379 19942 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786334794676589-31872-0/minicluster-data/ts-0/data/info.pb
I20260810 04:10:31.318845 31872 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskws89Io/build/release/bin/kudu as pid 19942
I20260810 04:10:31.318958 31872 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786334794676589-31872-0/minicluster-data/ts-0/wal/instance
I20260810 04:10:31.319187 31872 external_mini_cluster.cc:1428] Setting key 68a2d026841c27f23e418a3f5224c965
I20260810 04:10:31.320399 31872 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskws89Io/build/release/bin/kudu
/tmp/dist-test-taskws89Io/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786334794676589-31872-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786334794676589-31872-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786334794676589-31872-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786334794676589-31872-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.32.2:0
--local_ip_for_outbound_sockets=127.31.32.2
--webserver_interface=127.31.32.2
--webserver_port=0
--tserver_master_addrs=127.31.32.62:38155
--builtin_ntp_servers=127.31.32.20:40509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 04:10:31.324971 19920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:31.319197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.1:36295 (local address 127.31.32.62:38155)
0810 04:10:31.319287 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0810 04:10:31.319289 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:10:31.319663 (+ 374us) server_negotiation.cc:407] Connection header received
0810 04:10:31.320333 (+ 670us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:10:31.320334 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:10:31.320345 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:10:31.320394 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:10:31.321353 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:31.321877 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:31.322999 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:31.323393 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:31.323478 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:10:31.324582 (+ 1104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:10:31.324583 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:10:31.324584 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:10:31.324589 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:10:31.324608 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:10:31.324612 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:10:31.324687 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0810 04:10:31.324813 (+ 126us) server_negotiation.cc:299] Negotiation successful
0810 04:10:31.324845 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 04:10:31.325183 20089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:31.319197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:38155 (local address 127.31.32.1:36295)
0810 04:10:31.319513 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0810 04:10:31.319555 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 04:10:31.320170 (+ 615us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:10:31.320471 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:10:31.320476 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:10:31.320486 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:10:31.321207 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:10:31.321213 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:10:31.322037 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:10:31.322039 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:10:31.322871 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:10:31.322874 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:10:31.323511 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:10:31.323512 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:10:31.323629 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:10:31.324422 (+ 793us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 04:10:31.324422 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:10:31.324451 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 04:10:31.324475 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:10:31.324686 (+ 211us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:10:31.324689 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:10:31.324691 (+ 2us) client_negotiation.cc:769] Sending connection context
0810 04:10:31.324798 (+ 107us) client_negotiation.cc:240] Negotiation successful
0810 04:10:31.324897 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":220,"threads_started":1}
I20260810 04:10:31.325771 20087 heartbeater.cc:347] Connected to a master server at 127.31.32.62:38155
I20260810 04:10:31.325898 20087 heartbeater.cc:464] Registering TS with master...
I20260810 04:10:31.326124 20087 heartbeater.cc:511] Master 127.31.32.62:38155 requested a full tablet report, sending...
I20260810 04:10:31.326753 19876 ts_manager.cc:194] Registered new tserver with Master: 00cad55fe33f45f38e32199b882073a7 (127.31.32.1:33103)
I20260810 04:10:31.327404 19876 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.32.1:36295
W20260810 04:10:31.406574 20090 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:10:31.406826 20090 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:10:31.406864 20090 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:10:31.408002 20090 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:10:31.408077 20090 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:10:31.408124 20090 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.32.2
I20260810 04:10:31.409509 20090 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.32.20:40509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786334794676589-31872-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786334794676589-31872-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.32.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786334794676589-31872-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.32.2
--webserver_port=0
--tserver_master_addrs=127.31.32.62:38155
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20090
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.32.2
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786334794676589-31872-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:10:31.409791 20090 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:10:31.410109 20090 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:10:31.415455 20096 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:10:31.415450 20095 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:10:31.415468 20099 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:10:31.415649 20090 server_base.cc:1034] running on GCE node
I20260810 04:10:31.416028 20090 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:10:31.416352 20090 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:10:31.417538 20090 hybrid_clock.cc:648] HybridClock initialized: now 1786335031417525 us; error 50 us; skew 500 ppm
I20260810 04:10:31.418921 20090 webserver.cc:460] Webserver started at http://127.31.32.2:33785/ using document root <none> and password file <none>
I20260810 04:10:31.419186 20090 fs_manager.cc:359] Metadata directory not provided
I20260810 04:10:31.419265 20090 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:10:31.419426 20090 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:10:31.420615 20090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786334794676589-31872-0/minicluster-data/ts-1/data/instance:
uuid: "a98da0584a9e4e1ab29ec59bb323821a"
format_stamp: "Formatted at 2026-08-10 04:10:31 on dist-test-slave-8w9v"
server_key: "f264bd55784b85e97eb72d103f99234f"
server_key_iv: "a6a82963ce3f1e72210ab6715a36b5ec"
server_key_version: "encryptionkey@0"
I20260810 04:10:31.420989 20090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786334794676589-31872-0/minicluster-data/ts-1/wal/instance:
uuid: "a98da0584a9e4e1ab29ec59bb323821a"
format_stamp: "Formatted at 2026-08-10 04:10:31 on dist-test-slave-8w9v"
server_key: "f264bd55784b85e97eb72d103f99234f"
server_key_iv: "a6a82963ce3f1e72210ab6715a36b5ec"
server_key_version: "encryptionkey@0"
I20260810 04:10:31.422330 20090 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:10:31.423061 20105 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:10:31.423237 20090 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 04:10:31.423342 20090 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786334794676589-31872-0/minicluster-data/ts-1/data,/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786334794676589-31872-0/minicluster-data/ts-1/wal
uuid: "a98da0584a9e4e1ab29ec59bb323821a"
format_stamp: "Formatted at 2026-08-10 04:10:31 on dist-test-slave-8w9v"
server_key: "f264bd55784b85e97eb72d103f99234f"
server_key_iv: "a6a82963ce3f1e72210ab6715a36b5ec"
server_key_version: "encryptionkey@0"
I20260810 04:10:31.423444 20090 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786334794676589-31872-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786334794676589-31872-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786334794676589-31872-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:10:31.438330 20090 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:10:31.438663 20090 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:10:31.438827 20090 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:10:31.439077 20090 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:10:31.439421 20090 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:10:31.439481 20090 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:10:31.439524 20090 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:10:31.439548 20090 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:10:31.445297 20090 rpc_server.cc:307] RPC server started. Bound to: 127.31.32.2:37903
I20260810 04:10:31.445423 20237 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.32.2:37903 every 8 connection(s)
I20260810 04:10:31.445725 20090 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786334794676589-31872-0/minicluster-data/ts-1/data/info.pb
I20260810 04:10:31.446673 31872 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskws89Io/build/release/bin/kudu as pid 20090
I20260810 04:10:31.446794 31872 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786334794676589-31872-0/minicluster-data/ts-1/wal/instance
I20260810 04:10:31.447039 31872 external_mini_cluster.cc:1428] Setting key d84e977f5261afc3549d073a15b30965
I20260810 04:10:31.448238 31872 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskws89Io/build/release/bin/kudu
/tmp/dist-test-taskws89Io/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786334794676589-31872-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786334794676589-31872-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786334794676589-31872-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786334794676589-31872-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.32.3:0
--local_ip_for_outbound_sockets=127.31.32.3
--webserver_interface=127.31.32.3
--webserver_port=0
--tserver_master_addrs=127.31.32.62:38155
--builtin_ntp_servers=127.31.32.20:40509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 04:10:31.451779 19920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:31.446569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.2:56935 (local address 127.31.32.62:38155)
0810 04:10:31.446657 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0810 04:10:31.446660 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:10:31.446937 (+ 277us) server_negotiation.cc:407] Connection header received
0810 04:10:31.447574 (+ 637us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:10:31.447575 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:10:31.447586 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:10:31.447646 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:10:31.448514 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:31.449024 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:31.450351 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:31.450580 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:31.450666 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:10:31.451512 (+ 846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:10:31.451513 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:10:31.451513 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:10:31.451518 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:10:31.451533 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:10:31.451537 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:10:31.451583 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0810 04:10:31.451695 (+ 112us) server_negotiation.cc:299] Negotiation successful
0810 04:10:31.451720 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 04:10:31.452051 20240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:31.446472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:38155 (local address 127.31.32.2:56935)
0810 04:10:31.446777 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0810 04:10:31.446828 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0810 04:10:31.447479 (+ 651us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:10:31.447699 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:10:31.447702 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:10:31.447711 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:10:31.448427 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:10:31.448435 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:10:31.449131 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:10:31.449132 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:10:31.450244 (+ 1112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:10:31.450247 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:10:31.450678 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:10:31.450679 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:10:31.450813 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:10:31.451333 (+ 520us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 04:10:31.451333 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:10:31.451374 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 04:10:31.451411 (+ 37us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:10:31.451609 (+ 198us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:10:31.451615 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:10:31.451618 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 04:10:31.451688 (+ 70us) client_negotiation.cc:240] Negotiation successful
0810 04:10:31.451801 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":84,"threads_started":1}
I20260810 04:10:31.452546 20238 heartbeater.cc:347] Connected to a master server at 127.31.32.62:38155
I20260810 04:10:31.452672 20238 heartbeater.cc:464] Registering TS with master...
I20260810 04:10:31.452857 20238 heartbeater.cc:511] Master 127.31.32.62:38155 requested a full tablet report, sending...
I20260810 04:10:31.453307 19876 ts_manager.cc:194] Registered new tserver with Master: a98da0584a9e4e1ab29ec59bb323821a (127.31.32.2:37903)
I20260810 04:10:31.453678 19876 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.32.2:56935
W20260810 04:10:31.532953 20241 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:10:31.533160 20241 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:10:31.533182 20241 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:10:31.534333 20241 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:10:31.534376 20241 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:10:31.534415 20241 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.32.3
I20260810 04:10:31.535826 20241 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.32.20:40509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786334794676589-31872-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786334794676589-31872-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.32.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786334794676589-31872-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.32.3
--webserver_port=0
--tserver_master_addrs=127.31.32.62:38155
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20241
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.32.3
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786334794676589-31872-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:10:31.536181 20241 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:10:31.536528 20241 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:10:31.541694 20249 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:10:31.541880 20247 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:10:31.542025 20241 server_base.cc:1034] running on GCE node
W20260810 04:10:31.542039 20246 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:10:31.542307 20241 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:10:31.542580 20241 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:10:31.543773 20241 hybrid_clock.cc:648] HybridClock initialized: now 1786335031543759 us; error 51 us; skew 500 ppm
I20260810 04:10:31.545116 20241 webserver.cc:460] Webserver started at http://127.31.32.3:43923/ using document root <none> and password file <none>
I20260810 04:10:31.545405 20241 fs_manager.cc:359] Metadata directory not provided
I20260810 04:10:31.545487 20241 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:10:31.545653 20241 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:10:31.546728 20241 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786334794676589-31872-0/minicluster-data/ts-2/data/instance:
uuid: "77ea204c45604049ae961c4423f62298"
format_stamp: "Formatted at 2026-08-10 04:10:31 on dist-test-slave-8w9v"
server_key: "39050f3674c7b8767c9284e4ee821c25"
server_key_iv: "5ee9e145ac5d626977ddc75e33b8f874"
server_key_version: "encryptionkey@0"
I20260810 04:10:31.547128 20241 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786334794676589-31872-0/minicluster-data/ts-2/wal/instance:
uuid: "77ea204c45604049ae961c4423f62298"
format_stamp: "Formatted at 2026-08-10 04:10:31 on dist-test-slave-8w9v"
server_key: "39050f3674c7b8767c9284e4ee821c25"
server_key_iv: "5ee9e145ac5d626977ddc75e33b8f874"
server_key_version: "encryptionkey@0"
I20260810 04:10:31.548559 20241 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 04:10:31.549374 20256 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:10:31.549587 20241 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 04:10:31.549696 20241 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786334794676589-31872-0/minicluster-data/ts-2/data,/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786334794676589-31872-0/minicluster-data/ts-2/wal
uuid: "77ea204c45604049ae961c4423f62298"
format_stamp: "Formatted at 2026-08-10 04:10:31 on dist-test-slave-8w9v"
server_key: "39050f3674c7b8767c9284e4ee821c25"
server_key_iv: "5ee9e145ac5d626977ddc75e33b8f874"
server_key_version: "encryptionkey@0"
I20260810 04:10:31.549784 20241 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786334794676589-31872-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786334794676589-31872-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786334794676589-31872-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:10:31.566092 20241 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:10:31.566488 20241 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:10:31.566699 20241 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:10:31.566959 20241 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:10:31.567363 20241 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:10:31.567431 20241 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:10:31.567474 20241 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:10:31.567503 20241 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:10:31.573621 20241 rpc_server.cc:307] RPC server started. Bound to: 127.31.32.3:45859
I20260810 04:10:31.573671 20384 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.32.3:45859 every 8 connection(s)
I20260810 04:10:31.574024 20241 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786334794676589-31872-0/minicluster-data/ts-2/data/info.pb
I20260810 04:10:31.574561 31872 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskws89Io/build/release/bin/kudu as pid 20241
I20260810 04:10:31.574689 31872 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786334794676589-31872-0/minicluster-data/ts-2/wal/instance
I20260810 04:10:31.574889 31872 external_mini_cluster.cc:1428] Setting key 132f251c5eed925c56b8aecec4a8360f
I20260810 04:10:31.580461 19920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:31.575571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.3:57215 (local address 127.31.32.62:38155)
0810 04:10:31.575619 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0810 04:10:31.575621 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:10:31.575951 (+ 330us) server_negotiation.cc:407] Connection header received
0810 04:10:31.576623 (+ 672us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:10:31.576624 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:10:31.576635 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:10:31.576682 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:10:31.577394 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:31.577902 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:31.578918 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:31.579123 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:31.579214 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:10:31.580153 (+ 939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:10:31.580155 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:10:31.580155 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:10:31.580160 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:10:31.580175 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:10:31.580179 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:10:31.580248 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0810 04:10:31.580354 (+ 106us) server_negotiation.cc:299] Negotiation successful
0810 04:10:31.580382 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260810 04:10:31.580685 20389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:31.575533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:38155 (local address 127.31.32.3:57215)
0810 04:10:31.575783 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0810 04:10:31.575848 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0810 04:10:31.576509 (+ 661us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:10:31.576753 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:10:31.576757 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:10:31.576766 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:10:31.577294 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:10:31.577298 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:10:31.578007 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:10:31.578009 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:10:31.578800 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:10:31.578803 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:10:31.579227 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:10:31.579228 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:10:31.579381 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:10:31.579973 (+ 592us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 04:10:31.579974 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:10:31.580004 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 04:10:31.580028 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:10:31.580248 (+ 220us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:10:31.580255 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:10:31.580258 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 04:10:31.580331 (+ 73us) client_negotiation.cc:240] Negotiation successful
0810 04:10:31.580440 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":110,"threads_started":1}
I20260810 04:10:31.581173 20387 heartbeater.cc:347] Connected to a master server at 127.31.32.62:38155
I20260810 04:10:31.581255 20387 heartbeater.cc:464] Registering TS with master...
I20260810 04:10:31.581470 20387 heartbeater.cc:511] Master 127.31.32.62:38155 requested a full tablet report, sending...
I20260810 04:10:31.581843 19876 ts_manager.cc:194] Registered new tserver with Master: 77ea204c45604049ae961c4423f62298 (127.31.32.3:45859)
I20260810 04:10:31.582221 19876 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.32.3:57215
I20260810 04:10:31.587149 31872 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 04:10:31.591777 19920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:31.588691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51268 (local address 127.31.32.62:38155)
0810 04:10:31.588734 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0810 04:10:31.588735 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 04:10:31.588916 (+ 181us) server_negotiation.cc:407] Connection header received
0810 04:10:31.588990 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:10:31.588991 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:10:31.589002 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:10:31.589048 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:10:31.589233 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:31.589721 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:31.590500 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:31.590698 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:31.590789 (+ 91us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:10:31.591477 (+ 688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:10:31.591478 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:10:31.591479 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:10:31.591484 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:10:31.591497 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:10:31.591501 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:10:31.591615 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0810 04:10:31.591685 (+ 70us) server_negotiation.cc:299] Negotiation successful
0810 04:10:31.591697 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260810 04:10:31.593385 19876 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51268:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 04:10:31.595011 19876 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 04:10:31.601786 20397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:31.597550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59186 (local address 127.31.32.1:33103)
0810 04:10:31.597748 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0810 04:10:31.597753 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 04:10:31.597888 (+ 135us) server_negotiation.cc:407] Connection header received
0810 04:10:31.598018 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:10:31.598020 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:10:31.598034 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:10:31.598086 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:10:31.598350 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:31.599492 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:31.600807 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:31.601336 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:31.601459 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:10:31.601563 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0810 04:10:31.601642 (+ 79us) server_negotiation.cc:299] Negotiation successful
0810 04:10:31.601701 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":87,"threads_started":1}
I20260810 04:10:31.601761 20395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:31.597446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.1:33103 (local address 127.0.0.1:59186)
0810 04:10:31.597812 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0810 04:10:31.597823 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 04:10:31.597916 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:10:31.598178 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:10:31.598178 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:10:31.598180 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:10:31.598214 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:10:31.598216 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:10:31.599600 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:10:31.599600 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:10:31.600710 (+ 1110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:10:31.600714 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:10:31.601457 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:10:31.601458 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:10:31.601534 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:10:31.601547 (+ 13us) client_negotiation.cc:769] Sending connection context
0810 04:10:31.601627 (+ 80us) client_negotiation.cc:240] Negotiation successful
0810 04:10:31.601681 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":60,"threads_started":1}
I20260810 04:10:31.602164 20402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:31.597847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51444 (local address 127.31.32.3:45859)
0810 04:10:31.598031 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0810 04:10:31.598035 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 04:10:31.598044 (+ 9us) server_negotiation.cc:407] Connection header received
0810 04:10:31.598131 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:10:31.598133 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:10:31.598149 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:10:31.598231 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:10:31.598442 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:31.599221 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:31.600835 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:31.601566 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:31.601637 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:10:31.601720 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0810 04:10:31.602019 (+ 299us) server_negotiation.cc:299] Negotiation successful
0810 04:10:31.602066 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":83,"threads_started":1}
I20260810 04:10:31.602247 20398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:31.597626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58640 (local address 127.31.32.2:37903)
0810 04:10:31.597851 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0810 04:10:31.597855 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 04:10:31.597865 (+ 10us) server_negotiation.cc:407] Connection header received
0810 04:10:31.597910 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:10:31.597913 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:10:31.597929 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:10:31.598018 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:10:31.598254 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:31.599490 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:31.600835 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:31.601565 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:31.601638 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:10:31.601719 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0810 04:10:31.602120 (+ 401us) server_negotiation.cc:299] Negotiation successful
0810 04:10:31.602171 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":91,"threads_started":1}
I20260810 04:10:31.602035 20396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:31.597444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.2:37903 (local address 127.0.0.1:58640)
0810 04:10:31.597607 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0810 04:10:31.597647 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0810 04:10:31.597714 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:10:31.598100 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:10:31.598103 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:10:31.598105 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:10:31.598171 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:10:31.598173 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:10:31.599580 (+ 1407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:10:31.599581 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:10:31.600729 (+ 1148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:10:31.600731 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:10:31.601759 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:10:31.601760 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:10:31.601843 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:10:31.601847 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 04:10:31.601901 (+ 54us) client_negotiation.cc:240] Negotiation successful
0810 04:10:31.601946 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"mutex_wait_us":34,"thread_start_us":107,"threads_started":1}
I20260810 04:10:31.602914 20010 tablet_service.cc:1467] Processing CreateTablet for tablet 5964189d57734e5ea121832b483c66a9 (DEFAULT_TABLE table=test-table [id=f707f7425d6041ccae435045053925aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:10:31.603289 20010 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5964189d57734e5ea121832b483c66a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:10:31.603199 20160 tablet_service.cc:1467] Processing CreateTablet for tablet 5964189d57734e5ea121832b483c66a9 (DEFAULT_TABLE table=test-table [id=f707f7425d6041ccae435045053925aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:10:31.603534 20160 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5964189d57734e5ea121832b483c66a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:10:31.606299 20403 tablet_bootstrap.cc:492] T 5964189d57734e5ea121832b483c66a9 P 00cad55fe33f45f38e32199b882073a7: Bootstrap starting.
I20260810 04:10:31.606462 20401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:31.597661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.3:45859 (local address 127.0.0.1:51444)
0810 04:10:31.597875 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0810 04:10:31.597886 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 04:10:31.597943 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:10:31.598298 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:10:31.598299 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:10:31.598301 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:10:31.598346 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:10:31.598348 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:10:31.599339 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:10:31.599340 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:10:31.600729 (+ 1389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:10:31.600731 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:10:31.601761 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:10:31.601761 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:10:31.601943 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:10:31.601946 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 04:10:31.601998 (+ 52us) client_negotiation.cc:240] Negotiation successful
0810 04:10:31.602030 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"mutex_wait_us":6,"thread_start_us":71,"threads_started":1}
I20260810 04:10:31.606663 20405 tablet_bootstrap.cc:492] T 5964189d57734e5ea121832b483c66a9 P a98da0584a9e4e1ab29ec59bb323821a: Bootstrap starting.
I20260810 04:10:31.607053 20403 tablet_bootstrap.cc:654] T 5964189d57734e5ea121832b483c66a9 P 00cad55fe33f45f38e32199b882073a7: Neither blocks nor log segments found. Creating new log.
I20260810 04:10:31.607157 20311 tablet_service.cc:1467] Processing CreateTablet for tablet 5964189d57734e5ea121832b483c66a9 (DEFAULT_TABLE table=test-table [id=f707f7425d6041ccae435045053925aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:10:31.607308 20405 tablet_bootstrap.cc:654] T 5964189d57734e5ea121832b483c66a9 P a98da0584a9e4e1ab29ec59bb323821a: Neither blocks nor log segments found. Creating new log.
I20260810 04:10:31.607388 20403 log.cc:824] T 5964189d57734e5ea121832b483c66a9 P 00cad55fe33f45f38e32199b882073a7: Log is configured to *not* fsync() on all Append() calls
I20260810 04:10:31.607540 20311 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5964189d57734e5ea121832b483c66a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:10:31.607615 20405 log.cc:824] T 5964189d57734e5ea121832b483c66a9 P a98da0584a9e4e1ab29ec59bb323821a: Log is configured to *not* fsync() on all Append() calls
I20260810 04:10:31.608246 20403 tablet_bootstrap.cc:492] T 5964189d57734e5ea121832b483c66a9 P 00cad55fe33f45f38e32199b882073a7: No bootstrap required, opened a new log
I20260810 04:10:31.608320 20403 ts_tablet_manager.cc:1397] T 5964189d57734e5ea121832b483c66a9 P 00cad55fe33f45f38e32199b882073a7: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 04:10:31.608412 20405 tablet_bootstrap.cc:492] T 5964189d57734e5ea121832b483c66a9 P a98da0584a9e4e1ab29ec59bb323821a: No bootstrap required, opened a new log
I20260810 04:10:31.608477 20405 ts_tablet_manager.cc:1397] T 5964189d57734e5ea121832b483c66a9 P a98da0584a9e4e1ab29ec59bb323821a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 04:10:31.610387 20405 raft_consensus.cc:348] T 5964189d57734e5ea121832b483c66a9 P a98da0584a9e4e1ab29ec59bb323821a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00cad55fe33f45f38e32199b882073a7" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 33103 } } peers { permanent_uuid: "a98da0584a9e4e1ab29ec59bb323821a" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 37903 } } peers { permanent_uuid: "77ea204c45604049ae961c4423f62298" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 45859 } }
I20260810 04:10:31.610554 20405 raft_consensus.cc:374] T 5964189d57734e5ea121832b483c66a9 P a98da0584a9e4e1ab29ec59bb323821a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:10:31.610600 20405 raft_consensus.cc:729] T 5964189d57734e5ea121832b483c66a9 P a98da0584a9e4e1ab29ec59bb323821a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a98da0584a9e4e1ab29ec59bb323821a, State: Initialized, Role: FOLLOWER
I20260810 04:10:31.610716 20405 consensus_queue.cc:260] T 5964189d57734e5ea121832b483c66a9 P a98da0584a9e4e1ab29ec59bb323821a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00cad55fe33f45f38e32199b882073a7" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 33103 } } peers { permanent_uuid: "a98da0584a9e4e1ab29ec59bb323821a" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 37903 } } peers { permanent_uuid: "77ea204c45604049ae961c4423f62298" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 45859 } }
I20260810 04:10:31.611032 20405 ts_tablet_manager.cc:1428] T 5964189d57734e5ea121832b483c66a9 P a98da0584a9e4e1ab29ec59bb323821a: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260810 04:10:31.610942 20403 raft_consensus.cc:348] T 5964189d57734e5ea121832b483c66a9 P 00cad55fe33f45f38e32199b882073a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00cad55fe33f45f38e32199b882073a7" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 33103 } } peers { permanent_uuid: "a98da0584a9e4e1ab29ec59bb323821a" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 37903 } } peers { permanent_uuid: "77ea204c45604049ae961c4423f62298" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 45859 } }
I20260810 04:10:31.611045 20238 heartbeater.cc:503] Master 127.31.32.62:38155 was elected leader, sending a full tablet report...
I20260810 04:10:31.611097 20403 raft_consensus.cc:374] T 5964189d57734e5ea121832b483c66a9 P 00cad55fe33f45f38e32199b882073a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:10:31.611140 20403 raft_consensus.cc:729] T 5964189d57734e5ea121832b483c66a9 P 00cad55fe33f45f38e32199b882073a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00cad55fe33f45f38e32199b882073a7, State: Initialized, Role: FOLLOWER
I20260810 04:10:31.611268 20408 tablet_bootstrap.cc:492] T 5964189d57734e5ea121832b483c66a9 P 77ea204c45604049ae961c4423f62298: Bootstrap starting.
I20260810 04:10:31.611258 20403 consensus_queue.cc:260] T 5964189d57734e5ea121832b483c66a9 P 00cad55fe33f45f38e32199b882073a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00cad55fe33f45f38e32199b882073a7" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 33103 } } peers { permanent_uuid: "a98da0584a9e4e1ab29ec59bb323821a" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 37903 } } peers { permanent_uuid: "77ea204c45604049ae961c4423f62298" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 45859 } }
I20260810 04:10:31.611521 20403 ts_tablet_manager.cc:1428] T 5964189d57734e5ea121832b483c66a9 P 00cad55fe33f45f38e32199b882073a7: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 04:10:31.611652 20087 heartbeater.cc:503] Master 127.31.32.62:38155 was elected leader, sending a full tablet report...
I20260810 04:10:31.612270 20408 tablet_bootstrap.cc:654] T 5964189d57734e5ea121832b483c66a9 P 77ea204c45604049ae961c4423f62298: Neither blocks nor log segments found. Creating new log.
I20260810 04:10:31.612610 20408 log.cc:824] T 5964189d57734e5ea121832b483c66a9 P 77ea204c45604049ae961c4423f62298: Log is configured to *not* fsync() on all Append() calls
I20260810 04:10:31.613287 20408 tablet_bootstrap.cc:492] T 5964189d57734e5ea121832b483c66a9 P 77ea204c45604049ae961c4423f62298: No bootstrap required, opened a new log
I20260810 04:10:31.613361 20408 ts_tablet_manager.cc:1397] T 5964189d57734e5ea121832b483c66a9 P 77ea204c45604049ae961c4423f62298: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260810 04:10:31.615320 20408 raft_consensus.cc:348] T 5964189d57734e5ea121832b483c66a9 P 77ea204c45604049ae961c4423f62298 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00cad55fe33f45f38e32199b882073a7" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 33103 } } peers { permanent_uuid: "a98da0584a9e4e1ab29ec59bb323821a" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 37903 } } peers { permanent_uuid: "77ea204c45604049ae961c4423f62298" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 45859 } }
I20260810 04:10:31.615455 20408 raft_consensus.cc:374] T 5964189d57734e5ea121832b483c66a9 P 77ea204c45604049ae961c4423f62298 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:10:31.615482 20408 raft_consensus.cc:729] T 5964189d57734e5ea121832b483c66a9 P 77ea204c45604049ae961c4423f62298 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77ea204c45604049ae961c4423f62298, State: Initialized, Role: FOLLOWER
I20260810 04:10:31.615607 20408 consensus_queue.cc:260] T 5964189d57734e5ea121832b483c66a9 P 77ea204c45604049ae961c4423f62298 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00cad55fe33f45f38e32199b882073a7" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 33103 } } peers { permanent_uuid: "a98da0584a9e4e1ab29ec59bb323821a" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 37903 } } peers { permanent_uuid: "77ea204c45604049ae961c4423f62298" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 45859 } }
I20260810 04:10:31.615849 20408 ts_tablet_manager.cc:1428] T 5964189d57734e5ea121832b483c66a9 P 77ea204c45604049ae961c4423f62298: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260810 04:10:31.615949 20387 heartbeater.cc:503] Master 127.31.32.62:38155 was elected leader, sending a full tablet report...
W20260810 04:10:31.696337 20239 tablet.cc:2369] T 5964189d57734e5ea121832b483c66a9 P a98da0584a9e4e1ab29ec59bb323821a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 04:10:31.819875 20088 tablet.cc:2369] T 5964189d57734e5ea121832b483c66a9 P 00cad55fe33f45f38e32199b882073a7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 04:10:31.824597 20388 tablet.cc:2369] T 5964189d57734e5ea121832b483c66a9 P 77ea204c45604049ae961c4423f62298: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 04:10:31.885579 20412 raft_consensus.cc:482] T 5964189d57734e5ea121832b483c66a9 P 77ea204c45604049ae961c4423f62298 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 04:10:31.885727 20412 raft_consensus.cc:504] T 5964189d57734e5ea121832b483c66a9 P 77ea204c45604049ae961c4423f62298 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00cad55fe33f45f38e32199b882073a7" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 33103 } } peers { permanent_uuid: "a98da0584a9e4e1ab29ec59bb323821a" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 37903 } } peers { permanent_uuid: "77ea204c45604049ae961c4423f62298" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 45859 } }
I20260810 04:10:31.886266 20412 leader_election.cc:288] T 5964189d57734e5ea121832b483c66a9 P 77ea204c45604049ae961c4423f62298 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00cad55fe33f45f38e32199b882073a7 (127.31.32.1:33103), a98da0584a9e4e1ab29ec59bb323821a (127.31.32.2:37903)
I20260810 04:10:31.890204 20397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:31.886486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.3:48825 (local address 127.31.32.1:33103)
0810 04:10:31.886553 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0810 04:10:31.886555 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:10:31.886570 (+ 15us) server_negotiation.cc:407] Connection header received
0810 04:10:31.886606 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:10:31.886606 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:10:31.886621 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:10:31.886685 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:10:31.886923 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:31.887693 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:31.889226 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:31.889792 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:31.889883 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:10:31.889895 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0810 04:10:31.890091 (+ 196us) server_negotiation.cc:299] Negotiation successful
0810 04:10:31.890130 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260810 04:10:31.890204 20398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:31.886560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.3:47589 (local address 127.31.32.2:37903)
0810 04:10:31.886623 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0810 04:10:31.886625 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:10:31.886770 (+ 145us) server_negotiation.cc:407] Connection header received
0810 04:10:31.886816 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:10:31.886817 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:10:31.886826 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:10:31.886919 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:10:31.887081 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:31.887772 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:31.889226 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:31.889829 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:31.889893 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:10:31.889903 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0810 04:10:31.890091 (+ 188us) server_negotiation.cc:299] Negotiation successful
0810 04:10:31.890133 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260810 04:10:31.890205 20389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:31.886419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.1:33103 (local address 127.31.32.3:48825)
0810 04:10:31.886475 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0810 04:10:31.886489 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 04:10:31.886535 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:10:31.886733 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:10:31.886734 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:10:31.886735 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:10:31.886820 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:10:31.886822 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:10:31.887782 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:10:31.887782 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:10:31.889102 (+ 1320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:10:31.889105 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:10:31.889919 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:10:31.889920 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:10:31.889987 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:10:31.889992 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 04:10:31.890053 (+ 61us) client_negotiation.cc:240] Negotiation successful
0810 04:10:31.890116 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36,"mutex_wait_us":36}
I20260810 04:10:31.890213 20413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:31.886453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.2:37903 (local address 127.31.32.3:47589)
0810 04:10:31.886642 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0810 04:10:31.886652 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 04:10:31.886726 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:10:31.886949 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:10:31.886950 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:10:31.886956 (+ 6us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:10:31.887003 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:10:31.887004 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:10:31.887862 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:10:31.887862 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:10:31.889102 (+ 1240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:10:31.889105 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:10:31.889913 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:10:31.889914 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:10:31.889998 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:10:31.890001 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 04:10:31.890053 (+ 52us) client_negotiation.cc:240] Negotiation successful
0810 04:10:31.890114 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"mutex_wait_us":26,"thread_start_us":107,"threads_started":1}
I20260810 04:10:31.890708 20032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5964189d57734e5ea121832b483c66a9" candidate_uuid: "77ea204c45604049ae961c4423f62298" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00cad55fe33f45f38e32199b882073a7" is_pre_election: true
I20260810 04:10:31.890897 20032 raft_consensus.cc:2454] T 5964189d57734e5ea121832b483c66a9 P 00cad55fe33f45f38e32199b882073a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77ea204c45604049ae961c4423f62298 in term 0.
I20260810 04:10:31.891052 20183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5964189d57734e5ea121832b483c66a9" candidate_uuid: "77ea204c45604049ae961c4423f62298" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a98da0584a9e4e1ab29ec59bb323821a" is_pre_election: true
I20260810 04:10:31.891199 20183 raft_consensus.cc:2454] T 5964189d57734e5ea121832b483c66a9 P a98da0584a9e4e1ab29ec59bb323821a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77ea204c45604049ae961c4423f62298 in term 0.
I20260810 04:10:31.891198 20257 leader_election.cc:302] T 5964189d57734e5ea121832b483c66a9 P 77ea204c45604049ae961c4423f62298 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00cad55fe33f45f38e32199b882073a7, 77ea204c45604049ae961c4423f62298; no voters:
I20260810 04:10:31.891352 20412 raft_consensus.cc:2781] T 5964189d57734e5ea121832b483c66a9 P 77ea204c45604049ae961c4423f62298 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 04:10:31.891405 20412 raft_consensus.cc:482] T 5964189d57734e5ea121832b483c66a9 P 77ea204c45604049ae961c4423f62298 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 04:10:31.891433 20412 raft_consensus.cc:3037] T 5964189d57734e5ea121832b483c66a9 P 77ea204c45604049ae961c4423f62298 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:10:31.892261 20412 raft_consensus.cc:504] T 5964189d57734e5ea121832b483c66a9 P 77ea204c45604049ae961c4423f62298 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00cad55fe33f45f38e32199b882073a7" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 33103 } } peers { permanent_uuid: "a98da0584a9e4e1ab29ec59bb323821a" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 37903 } } peers { permanent_uuid: "77ea204c45604049ae961c4423f62298" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 45859 } }
I20260810 04:10:31.892424 20412 leader_election.cc:288] T 5964189d57734e5ea121832b483c66a9 P 77ea204c45604049ae961c4423f62298 [CANDIDATE]: Term 1 election: Requested vote from peers 00cad55fe33f45f38e32199b882073a7 (127.31.32.1:33103), a98da0584a9e4e1ab29ec59bb323821a (127.31.32.2:37903)
I20260810 04:10:31.892577 20032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5964189d57734e5ea121832b483c66a9" candidate_uuid: "77ea204c45604049ae961c4423f62298" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00cad55fe33f45f38e32199b882073a7"
I20260810 04:10:31.892589 20183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5964189d57734e5ea121832b483c66a9" candidate_uuid: "77ea204c45604049ae961c4423f62298" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a98da0584a9e4e1ab29ec59bb323821a"
I20260810 04:10:31.892676 20032 raft_consensus.cc:3037] T 5964189d57734e5ea121832b483c66a9 P 00cad55fe33f45f38e32199b882073a7 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:10:31.892673 20183 raft_consensus.cc:3037] T 5964189d57734e5ea121832b483c66a9 P a98da0584a9e4e1ab29ec59bb323821a [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:10:31.893383 20032 raft_consensus.cc:2454] T 5964189d57734e5ea121832b483c66a9 P 00cad55fe33f45f38e32199b882073a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77ea204c45604049ae961c4423f62298 in term 1.
I20260810 04:10:31.893453 20183 raft_consensus.cc:2454] T 5964189d57734e5ea121832b483c66a9 P a98da0584a9e4e1ab29ec59bb323821a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77ea204c45604049ae961c4423f62298 in term 1.
I20260810 04:10:31.893612 20257 leader_election.cc:302] T 5964189d57734e5ea121832b483c66a9 P 77ea204c45604049ae961c4423f62298 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00cad55fe33f45f38e32199b882073a7, 77ea204c45604049ae961c4423f62298; no voters:
I20260810 04:10:31.893698 20412 raft_consensus.cc:2781] T 5964189d57734e5ea121832b483c66a9 P 77ea204c45604049ae961c4423f62298 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:10:31.893862 20412 raft_consensus.cc:686] T 5964189d57734e5ea121832b483c66a9 P 77ea204c45604049ae961c4423f62298 [term 1 LEADER]: Becoming Leader. State: Replica: 77ea204c45604049ae961c4423f62298, State: Running, Role: LEADER
I20260810 04:10:31.893971 20412 consensus_queue.cc:237] T 5964189d57734e5ea121832b483c66a9 P 77ea204c45604049ae961c4423f62298 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00cad55fe33f45f38e32199b882073a7" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 33103 } } peers { permanent_uuid: "a98da0584a9e4e1ab29ec59bb323821a" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 37903 } } peers { permanent_uuid: "77ea204c45604049ae961c4423f62298" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 45859 } }
I20260810 04:10:31.894871 19875 catalog_manager.cc:5495] T 5964189d57734e5ea121832b483c66a9 P 77ea204c45604049ae961c4423f62298 reported cstate change: term changed from 0 to 1, leader changed from <none> to 77ea204c45604049ae961c4423f62298 (127.31.32.3). New cstate: current_term: 1 leader_uuid: "77ea204c45604049ae961c4423f62298" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00cad55fe33f45f38e32199b882073a7" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 33103 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a98da0584a9e4e1ab29ec59bb323821a" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 37903 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77ea204c45604049ae961c4423f62298" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 45859 } health_report { overall_health: HEALTHY } } }
I20260810 04:10:31.955955 20402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:31.953422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51448 (local address 127.31.32.3:45859)
0810 04:10:31.953510 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0810 04:10:31.953512 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:10:31.953524 (+ 12us) server_negotiation.cc:407] Connection header received
0810 04:10:31.953625 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:10:31.953626 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:10:31.953648 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:10:31.953715 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 04:10:31.953868 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:31.954330 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:31.955155 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:31.955350 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:31.955412 (+ 62us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:10:31.955633 (+ 221us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 04:10:31.955728 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 04:10:31.955815 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0810 04:10:31.955868 (+ 53us) server_negotiation.cc:299] Negotiation successful
0810 04:10:31.955875 (+ 7us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 04:10:31.957973 20183 raft_consensus.cc:1261] T 5964189d57734e5ea121832b483c66a9 P a98da0584a9e4e1ab29ec59bb323821a [term 1 FOLLOWER]: Refusing update from remote peer 77ea204c45604049ae961c4423f62298: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:10:31.958334 20032 raft_consensus.cc:1261] T 5964189d57734e5ea121832b483c66a9 P 00cad55fe33f45f38e32199b882073a7 [term 1 FOLLOWER]: Refusing update from remote peer 77ea204c45604049ae961c4423f62298: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:10:31.958388 20414 consensus_queue.cc:1035] T 5964189d57734e5ea121832b483c66a9 P 77ea204c45604049ae961c4423f62298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a98da0584a9e4e1ab29ec59bb323821a" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 37903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:10:31.958710 20414 consensus_queue.cc:1035] T 5964189d57734e5ea121832b483c66a9 P 77ea204c45604049ae961c4423f62298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00cad55fe33f45f38e32199b882073a7" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 33103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:10:31.960847 20417 mvcc.cc:205] Tried to move back new op lower bound from 7316828290896777216 to 7316828290638274560. Current Snapshot: MvccSnapshot[applied={T|T < 7316828290896777216}]
I20260810 04:10:31.961309 20418 mvcc.cc:205] Tried to move back new op lower bound from 7316828290896777216 to 7316828290638274560. Current Snapshot: MvccSnapshot[applied={T|T < 7316828290896777216}]
I20260810 04:10:31.963356 19876 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51268:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 04:10:31.963471 19876 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51268:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 04:10:31.964352 19876 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 80339afd93d945e08e147eff38c7fe7c: Sending DeleteTablet for 3 replicas of tablet 5964189d57734e5ea121832b483c66a9
I20260810 04:10:31.964676 20010 tablet_service.cc:1514] Processing DeleteTablet for tablet 5964189d57734e5ea121832b483c66a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:10:31 UTC) from {username='slave'} at 127.0.0.1:59186
I20260810 04:10:31.964701 20160 tablet_service.cc:1514] Processing DeleteTablet for tablet 5964189d57734e5ea121832b483c66a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:10:31 UTC) from {username='slave'} at 127.0.0.1:58640
I20260810 04:10:31.964980 20427 tablet_replica.cc:330] T 5964189d57734e5ea121832b483c66a9 P a98da0584a9e4e1ab29ec59bb323821a: stopping tablet replica
I20260810 04:10:31.964963 20311 tablet_service.cc:1514] Processing DeleteTablet for tablet 5964189d57734e5ea121832b483c66a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:10:31 UTC) from {username='slave'} at 127.0.0.1:51444
I20260810 04:10:31.965082 20427 raft_consensus.cc:2229] T 5964189d57734e5ea121832b483c66a9 P a98da0584a9e4e1ab29ec59bb323821a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 04:10:31.965190 20427 raft_consensus.cc:2258] T 5964189d57734e5ea121832b483c66a9 P a98da0584a9e4e1ab29ec59bb323821a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:10:31.965185 31872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskws89Io/build/release/bin/kudu with pid 19942
I20260810 04:10:31.966821 20427 ts_tablet_manager.cc:1905] T 5964189d57734e5ea121832b483c66a9 P a98da0584a9e4e1ab29ec59bb323821a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:10:31.969201 20427 ts_tablet_manager.cc:1918] T 5964189d57734e5ea121832b483c66a9 P a98da0584a9e4e1ab29ec59bb323821a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 04:10:31.969324 20427 log.cc:1196] T 5964189d57734e5ea121832b483c66a9 P a98da0584a9e4e1ab29ec59bb323821a: Deleting WAL directory at /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786334794676589-31872-0/minicluster-data/ts-1/wal/wals/5964189d57734e5ea121832b483c66a9
I20260810 04:10:31.969626 20427 ts_tablet_manager.cc:1939] T 5964189d57734e5ea121832b483c66a9 P a98da0584a9e4e1ab29ec59bb323821a: Deleting consensus metadata
I20260810 04:10:31.970043 19860 catalog_manager.cc:4841] TS a98da0584a9e4e1ab29ec59bb323821a (127.31.32.2:37903): tablet 5964189d57734e5ea121832b483c66a9 (table test-table [id=f707f7425d6041ccae435045053925aa]) successfully deleted
I20260810 04:10:31.972415 20429 tablet_replica.cc:330] T 5964189d57734e5ea121832b483c66a9 P 77ea204c45604049ae961c4423f62298: stopping tablet replica
I20260810 04:10:31.972559 20429 raft_consensus.cc:2229] T 5964189d57734e5ea121832b483c66a9 P 77ea204c45604049ae961c4423f62298 [term 1 LEADER]: Raft consensus shutting down.
I20260810 04:10:31.972675 20429 raft_consensus.cc:2258] T 5964189d57734e5ea121832b483c66a9 P 77ea204c45604049ae961c4423f62298 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:10:31.973100 20429 ts_tablet_manager.cc:1905] T 5964189d57734e5ea121832b483c66a9 P 77ea204c45604049ae961c4423f62298: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260810 04:10:31.974684 19859 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.32.1:33103 (error 108)
I20260810 04:10:31.974946 31872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskws89Io/build/release/bin/kudu with pid 20090
I20260810 04:10:31.975106 20429 ts_tablet_manager.cc:1918] T 5964189d57734e5ea121832b483c66a9 P 77ea204c45604049ae961c4423f62298: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 04:10:31.975183 20429 log.cc:1196] T 5964189d57734e5ea121832b483c66a9 P 77ea204c45604049ae961c4423f62298: Deleting WAL directory at /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786334794676589-31872-0/minicluster-data/ts-2/wal/wals/5964189d57734e5ea121832b483c66a9
I20260810 04:10:31.975214 20401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:31.975047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.1:33103 (local address 127.0.0.1:59200)
0810 04:10:31.975130 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0810 04:10:31.975178 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.1:33103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260810 04:10:31.975471 20429 ts_tablet_manager.cc:1939] T 5964189d57734e5ea121832b483c66a9 P 77ea204c45604049ae961c4423f62298: Deleting consensus metadata
W20260810 04:10:31.975523 19859 catalog_manager.cc:4584] TS 00cad55fe33f45f38e32199b882073a7 (127.31.32.1:33103): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5964189d57734e5ea121832b483c66a9: Network error: Client connection negotiation failed: client connection to 127.31.32.1:33103: connect: Connection refused (error 111)
I20260810 04:10:31.975867 19860 catalog_manager.cc:4841] TS 77ea204c45604049ae961c4423f62298 (127.31.32.3:45859): tablet 5964189d57734e5ea121832b483c66a9 (table test-table [id=f707f7425d6041ccae435045053925aa]) successfully deleted
I20260810 04:10:31.985041 31872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskws89Io/build/release/bin/kudu with pid 20241
I20260810 04:10:31.991860 31872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskws89Io/build/release/bin/kudu with pid 19841
2026-08-10T04:10:31Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (998 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786334794676589-31872-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 04:10:32 dist-test-slave-8w9v krb5kdc[20433](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 04:10:32 dist-test-slave-8w9v krb5kdc[20433](info): set up 2 sockets
Aug 10 04:10:32 dist-test-slave-8w9v krb5kdc[20433](info): commencing operation
krb5kdc: starting...
W20260810 04:10:33.093205 31872 mini_kdc.cc:121] Time spent starting KDC: real 1.080s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786334794676589-31872-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786334794676589-31872-0/krb5kdc/test-user.keytab.
Aug 10 04:10:33 dist-test-slave-8w9v krb5kdc[20433](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335033, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T04:10:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T04:10:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.32.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.32.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.32.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.62.keytab.
Entry for principal kudu/127.31.32.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.62.keytab.
I20260810 04:10:33.227511 31872 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskws89Io/build/release/bin/kudu
/tmp/dist-test-taskws89Io/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786334794676589-31872-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786334794676589-31872-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786334794676589-31872-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786334794676589-31872-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.32.62:38367
--webserver_interface=127.31.32.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.62.keytab
--principal=kudu/127.31.32.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.32.20:39859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.32.62:38367
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786334794676589-31872-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786334794676589-31872-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786334794676589-31872-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:10:33.321492 20447 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:10:33.321702 20447 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:10:33.321724 20447 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:10:33.322901 20447 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:10:33.322945 20447 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:10:33.322960 20447 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:10:33.322973 20447 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:10:33.323041 20447 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:10:33.324544 20447 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.32.20:39859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786334794676589-31872-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786334794676589-31872-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.32.62:38367
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.62.keytab
--principal=kudu/127.31.32.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.32.62:38367
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786334794676589-31872-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.32.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20447
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786334794676589-31872-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:10:33.324820 20447 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:10:33.325081 20447 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:10:33.330112 20454 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:10:33.330127 20452 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:10:33.330299 20447 server_base.cc:1034] running on GCE node
W20260810 04:10:33.330121 20456 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:10:33.330643 20447 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:10:33.330992 20447 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:10:33.332206 20447 hybrid_clock.cc:648] HybridClock initialized: now 1786335033332185 us; error 51 us; skew 500 ppm
Aug 10 04:10:33 dist-test-slave-8w9v krb5kdc[20433](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335033, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:10:33.334707 20447 init.cc:377] Logged in from keytab as kudu/127.31.32.62@KRBTEST.COM (short username kudu)
I20260810 04:10:33.335220 20447 webserver.cc:460] Webserver started at http://127.31.32.62:40195/ using document root <none> and password file <none>
I20260810 04:10:33.335500 20447 fs_manager.cc:359] Metadata directory not provided
I20260810 04:10:33.335577 20447 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:10:33.335734 20447 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:10:33.336892 20447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786334794676589-31872-0/minicluster-data/master-0/data/instance:
uuid: "19f71e13a69a4503b21addf1c8a02317"
format_stamp: "Formatted at 2026-08-10 04:10:33 on dist-test-slave-8w9v"
server_key: "5ffa10514f37357743a49c64f3d19d04"
server_key_iv: "251207afb6b0cbff65194e8f524022f2"
server_key_version: "encryptionkey@0"
I20260810 04:10:33.337273 20447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786334794676589-31872-0/minicluster-data/master-0/wal/instance:
uuid: "19f71e13a69a4503b21addf1c8a02317"
format_stamp: "Formatted at 2026-08-10 04:10:33 on dist-test-slave-8w9v"
server_key: "5ffa10514f37357743a49c64f3d19d04"
server_key_iv: "251207afb6b0cbff65194e8f524022f2"
server_key_version: "encryptionkey@0"
I20260810 04:10:33.338686 20447 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 04:10:33.339493 20466 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:10:33.339802 20447 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:10:33.339918 20447 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786334794676589-31872-0/minicluster-data/master-0/data,/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786334794676589-31872-0/minicluster-data/master-0/wal
uuid: "19f71e13a69a4503b21addf1c8a02317"
format_stamp: "Formatted at 2026-08-10 04:10:33 on dist-test-slave-8w9v"
server_key: "5ffa10514f37357743a49c64f3d19d04"
server_key_iv: "251207afb6b0cbff65194e8f524022f2"
server_key_version: "encryptionkey@0"
I20260810 04:10:33.340003 20447 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786334794676589-31872-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786334794676589-31872-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786334794676589-31872-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:10:33.369766 20447 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:10:33.371901 20447 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:10:33.372087 20447 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:10:33.376500 20447 rpc_server.cc:307] RPC server started. Bound to: 127.31.32.62:38367
I20260810 04:10:33.376540 20521 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.32.62:38367 every 8 connection(s)
I20260810 04:10:33.377076 20447 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786334794676589-31872-0/minicluster-data/master-0/data/info.pb
I20260810 04:10:33.377775 20522 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:10:33.380234 20522 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19f71e13a69a4503b21addf1c8a02317: Bootstrap starting.
I20260810 04:10:33.380861 20522 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 19f71e13a69a4503b21addf1c8a02317: Neither blocks nor log segments found. Creating new log.
I20260810 04:10:33.381176 20522 log.cc:824] T 00000000000000000000000000000000 P 19f71e13a69a4503b21addf1c8a02317: Log is configured to *not* fsync() on all Append() calls
I20260810 04:10:33.381919 20522 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19f71e13a69a4503b21addf1c8a02317: No bootstrap required, opened a new log
I20260810 04:10:33.382328 31872 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskws89Io/build/release/bin/kudu as pid 20447
I20260810 04:10:33.382467 31872 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786334794676589-31872-0/minicluster-data/master-0/wal/instance
I20260810 04:10:33.382679 31872 external_mini_cluster.cc:1428] Setting key 75d03a7b651d1f5d698eb64ed9fbb72e
I20260810 04:10:33.384524 20522 raft_consensus.cc:348] T 00000000000000000000000000000000 P 19f71e13a69a4503b21addf1c8a02317 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19f71e13a69a4503b21addf1c8a02317" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 38367 } }
I20260810 04:10:33.384713 20522 raft_consensus.cc:374] T 00000000000000000000000000000000 P 19f71e13a69a4503b21addf1c8a02317 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:10:33.384752 20522 raft_consensus.cc:729] T 00000000000000000000000000000000 P 19f71e13a69a4503b21addf1c8a02317 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19f71e13a69a4503b21addf1c8a02317, State: Initialized, Role: FOLLOWER
I20260810 04:10:33.384892 20522 consensus_queue.cc:260] T 00000000000000000000000000000000 P 19f71e13a69a4503b21addf1c8a02317 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19f71e13a69a4503b21addf1c8a02317" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 38367 } }
I20260810 04:10:33.384984 20522 raft_consensus.cc:388] T 00000000000000000000000000000000 P 19f71e13a69a4503b21addf1c8a02317 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 04:10:33.385018 20522 raft_consensus.cc:482] T 00000000000000000000000000000000 P 19f71e13a69a4503b21addf1c8a02317 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 04:10:33.385082 20522 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 19f71e13a69a4503b21addf1c8a02317 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:10:33.385669 20522 raft_consensus.cc:504] T 00000000000000000000000000000000 P 19f71e13a69a4503b21addf1c8a02317 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19f71e13a69a4503b21addf1c8a02317" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 38367 } }
I20260810 04:10:33.385758 20522 leader_election.cc:302] T 00000000000000000000000000000000 P 19f71e13a69a4503b21addf1c8a02317 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 19f71e13a69a4503b21addf1c8a02317; no voters:
I20260810 04:10:33.385958 20522 leader_election.cc:288] T 00000000000000000000000000000000 P 19f71e13a69a4503b21addf1c8a02317 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 04:10:33.386009 20528 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 19f71e13a69a4503b21addf1c8a02317 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:10:33.386169 20528 raft_consensus.cc:686] T 00000000000000000000000000000000 P 19f71e13a69a4503b21addf1c8a02317 [term 1 LEADER]: Becoming Leader. State: Replica: 19f71e13a69a4503b21addf1c8a02317, State: Running, Role: LEADER
I20260810 04:10:33.386234 20522 sys_catalog.cc:564] T 00000000000000000000000000000000 P 19f71e13a69a4503b21addf1c8a02317 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:10:33.386308 20528 consensus_queue.cc:237] T 00000000000000000000000000000000 P 19f71e13a69a4503b21addf1c8a02317 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19f71e13a69a4503b21addf1c8a02317" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 38367 } }
I20260810 04:10:33.386615 20528 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19f71e13a69a4503b21addf1c8a02317 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 19f71e13a69a4503b21addf1c8a02317. Latest consensus state: current_term: 1 leader_uuid: "19f71e13a69a4503b21addf1c8a02317" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19f71e13a69a4503b21addf1c8a02317" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 38367 } } }
I20260810 04:10:33.386698 20528 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19f71e13a69a4503b21addf1c8a02317 [sys.catalog]: This master's current role is: LEADER
I20260810 04:10:33.386812 20529 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19f71e13a69a4503b21addf1c8a02317 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "19f71e13a69a4503b21addf1c8a02317" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19f71e13a69a4503b21addf1c8a02317" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 38367 } } }
I20260810 04:10:33.386961 20529 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19f71e13a69a4503b21addf1c8a02317 [sys.catalog]: This master's current role is: LEADER
W20260810 04:10:33.388204 20545 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 19f71e13a69a4503b21addf1c8a02317: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 04:10:33.388257 20545 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 04:10:33.388732 20535 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 04:10:33.388928 20535 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 04:10:33.390731 20535 catalog_manager.cc:1347] Generated new cluster ID: 234d9eabfb6d4d5c8637359a78b7e45b
I20260810 04:10:33.390791 20535 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 10 04:10:33 dist-test-slave-8w9v krb5kdc[20433](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335033, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.32.62@KRBTEST.COM
I20260810 04:10:33.395283 20527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:33.383237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56574 (local address 127.31.32.62:38367)
0810 04:10:33.383433 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0810 04:10:33.383440 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 04:10:33.383466 (+ 26us) server_negotiation.cc:407] Connection header received
0810 04:10:33.383982 (+ 516us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:10:33.383988 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:10:33.384034 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:10:33.384239 (+ 205us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:10:33.385041 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:33.386722 (+ 1681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:33.387805 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:33.388414 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:33.388543 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:10:33.391402 (+ 2859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:10:33.391406 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:10:33.391407 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:10:33.391420 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:10:33.393556 (+ 2136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:10:33.393954 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:10:33.393957 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:10:33.393958 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:10:33.394072 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:10:33.394287 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:10:33.394288 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:10:33.394288 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:10:33.394644 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:10:33.394833 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0810 04:10:33.394925 (+ 92us) server_negotiation.cc:299] Negotiation successful
0810 04:10:33.395070 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":93,"threads_started":1}
I20260810 04:10:33.401697 20535 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 04:10:33.402390 20535 catalog_manager.cc:1504] Loading token signing keys...
I20260810 04:10:33.405730 20535 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 19f71e13a69a4503b21addf1c8a02317: Generated new TSK 0
I20260810 04:10:33.405916 20535 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.32.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.32.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.32.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.1.keytab.
Entry for principal kudu/127.31.32.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.1.keytab.
I20260810 04:10:33.435456 31872 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskws89Io/build/release/bin/kudu
/tmp/dist-test-taskws89Io/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786334794676589-31872-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786334794676589-31872-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786334794676589-31872-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786334794676589-31872-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.32.1:0
--local_ip_for_outbound_sockets=127.31.32.1
--webserver_interface=127.31.32.1
--webserver_port=0
--tserver_master_addrs=127.31.32.62:38367
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.1.keytab
--principal=kudu/127.31.32.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.32.20:39859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786334794676589-31872-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786334794676589-31872-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786334794676589-31872-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:10:33.519721 20551 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:10:33.519918 20551 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:10:33.519941 20551 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:10:33.521131 20551 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:10:33.521176 20551 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:10:33.521202 20551 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.32.1
I20260810 04:10:33.522624 20551 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.32.20:39859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786334794676589-31872-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786334794676589-31872-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.1.keytab
--principal=kudu/127.31.32.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.32.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786334794676589-31872-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.32.1
--webserver_port=0
--tserver_master_addrs=127.31.32.62:38367
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20551
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.32.1
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786334794676589-31872-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:10:33.522822 20551 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:10:33.522998 20551 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:10:33.528220 20556 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:10:33.528326 20551 server_base.cc:1034] running on GCE node
W20260810 04:10:33.528260 20560 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:10:33.528221 20558 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:10:33.528683 20551 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:10:33.528944 20551 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:10:33.530114 20551 hybrid_clock.cc:648] HybridClock initialized: now 1786335033530081 us; error 49 us; skew 500 ppm
Aug 10 04:10:33 dist-test-slave-8w9v krb5kdc[20433](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335033, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:10:33.532851 20551 init.cc:377] Logged in from keytab as kudu/127.31.32.1@KRBTEST.COM (short username kudu)
I20260810 04:10:33.533285 20551 webserver.cc:460] Webserver started at http://127.31.32.1:41027/ using document root <none> and password file <none>
I20260810 04:10:33.533572 20551 fs_manager.cc:359] Metadata directory not provided
I20260810 04:10:33.533650 20551 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:10:33.533810 20551 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:10:33.534878 20551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786334794676589-31872-0/minicluster-data/ts-0/data/instance:
uuid: "02017577ae6045c58079adb7ea803efa"
format_stamp: "Formatted at 2026-08-10 04:10:33 on dist-test-slave-8w9v"
server_key: "afd8a6cf896e1ba6f94b17ee48fe747c"
server_key_iv: "e1722f9ab612e5808edc2b7718c1d232"
server_key_version: "encryptionkey@0"
I20260810 04:10:33.535251 20551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786334794676589-31872-0/minicluster-data/ts-0/wal/instance:
uuid: "02017577ae6045c58079adb7ea803efa"
format_stamp: "Formatted at 2026-08-10 04:10:33 on dist-test-slave-8w9v"
server_key: "afd8a6cf896e1ba6f94b17ee48fe747c"
server_key_iv: "e1722f9ab612e5808edc2b7718c1d232"
server_key_version: "encryptionkey@0"
I20260810 04:10:33.536823 20551 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 04:10:33.537708 20567 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:10:33.537937 20551 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:10:33.538059 20551 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786334794676589-31872-0/minicluster-data/ts-0/data,/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786334794676589-31872-0/minicluster-data/ts-0/wal
uuid: "02017577ae6045c58079adb7ea803efa"
format_stamp: "Formatted at 2026-08-10 04:10:33 on dist-test-slave-8w9v"
server_key: "afd8a6cf896e1ba6f94b17ee48fe747c"
server_key_iv: "e1722f9ab612e5808edc2b7718c1d232"
server_key_version: "encryptionkey@0"
I20260810 04:10:33.538144 20551 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786334794676589-31872-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786334794676589-31872-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786334794676589-31872-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:10:33.549597 20551 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:10:33.551723 20551 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:10:33.551898 20551 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:10:33.552181 20551 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:10:33.552508 20551 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:10:33.552570 20551 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:10:33.552604 20551 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:10:33.552632 20551 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:10:33.558432 20551 rpc_server.cc:307] RPC server started. Bound to: 127.31.32.1:39175
I20260810 04:10:33.558509 20692 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.32.1:39175 every 8 connection(s)
I20260810 04:10:33.558831 20551 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786334794676589-31872-0/minicluster-data/ts-0/data/info.pb
I20260810 04:10:33.560957 31872 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskws89Io/build/release/bin/kudu as pid 20551
I20260810 04:10:33.561069 31872 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786334794676589-31872-0/minicluster-data/ts-0/wal/instance
I20260810 04:10:33.561290 31872 external_mini_cluster.cc:1428] Setting key 85f28ce5a344318cd3613dc462d45e56
Aug 10 04:10:33 dist-test-slave-8w9v krb5kdc[20433](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335033, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.1@KRBTEST.COM for kudu/127.31.32.62@KRBTEST.COM
I20260810 04:10:33.569183 20527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:33.559659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.1:32971 (local address 127.31.32.62:38367)
0810 04:10:33.559749 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0810 04:10:33.559751 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:10:33.559921 (+ 170us) server_negotiation.cc:407] Connection header received
0810 04:10:33.560441 (+ 520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:10:33.560442 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:10:33.560453 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:10:33.560504 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:10:33.561680 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:33.562413 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:33.563388 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:33.563714 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:33.563790 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:10:33.566217 (+ 2427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:10:33.566219 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:10:33.566219 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:10:33.566227 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:10:33.567699 (+ 1472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:10:33.568169 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:10:33.568170 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:10:33.568171 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:10:33.568232 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:10:33.568503 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:10:33.568504 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:10:33.568504 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:10:33.568620 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:10:33.568700 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0810 04:10:33.568999 (+ 299us) server_negotiation.cc:299] Negotiation successful
0810 04:10:33.569089 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 04:10:33.569358 20696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:33.559602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:38367 (local address 127.31.32.1:32971)
0810 04:10:33.559797 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0810 04:10:33.559835 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0810 04:10:33.560349 (+ 514us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:10:33.560564 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:10:33.560566 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:10:33.561252 (+ 686us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:10:33.561571 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:10:33.561578 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:10:33.562510 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:10:33.562511 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:10:33.563287 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:10:33.563289 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:10:33.563821 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:10:33.563823 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:10:33.563999 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:10:33.564564 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:10:33.564574 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:10:33.566115 (+ 1541us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:10:33.567793 (+ 1678us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:10:33.567800 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:10:33.567803 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:10:33.568037 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:10:33.568315 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:10:33.568316 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:10:33.568316 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:10:33.568429 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:10:33.568706 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:10:33.568713 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:10:33.568858 (+ 145us) client_negotiation.cc:769] Sending connection context
0810 04:10:33.568964 (+ 106us) client_negotiation.cc:240] Negotiation successful
0810 04:10:33.569123 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":101,"threads_started":1}
I20260810 04:10:33.569875 20694 heartbeater.cc:347] Connected to a master server at 127.31.32.62:38367
I20260810 04:10:33.569990 20694 heartbeater.cc:464] Registering TS with master...
I20260810 04:10:33.570189 20694 heartbeater.cc:511] Master 127.31.32.62:38367 requested a full tablet report, sending...
I20260810 04:10:33.570732 20483 ts_manager.cc:194] Registered new tserver with Master: 02017577ae6045c58079adb7ea803efa (127.31.32.1:39175)
WARNING: no policy specified for kudu/127.31.32.2@KRBTEST.COM; defaulting to no policy
I20260810 04:10:33.571429 20483 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.32.1@KRBTEST.COM'} at 127.31.32.1:32971
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.32.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.32.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.2.keytab.
Entry for principal kudu/127.31.32.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.2.keytab.
I20260810 04:10:33.591295 31872 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskws89Io/build/release/bin/kudu
/tmp/dist-test-taskws89Io/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786334794676589-31872-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786334794676589-31872-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786334794676589-31872-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786334794676589-31872-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.32.2:0
--local_ip_for_outbound_sockets=127.31.32.2
--webserver_interface=127.31.32.2
--webserver_port=0
--tserver_master_addrs=127.31.32.62:38367
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.2.keytab
--principal=kudu/127.31.32.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.32.20:39859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786334794676589-31872-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786334794676589-31872-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786334794676589-31872-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:10:33.674139 20699 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:10:33.674394 20699 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:10:33.674435 20699 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:10:33.675590 20699 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:10:33.675665 20699 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:10:33.675710 20699 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.32.2
I20260810 04:10:33.677214 20699 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.32.20:39859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786334794676589-31872-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786334794676589-31872-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.2.keytab
--principal=kudu/127.31.32.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.32.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786334794676589-31872-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.32.2
--webserver_port=0
--tserver_master_addrs=127.31.32.62:38367
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20699
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.32.2
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786334794676589-31872-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:10:33.677495 20699 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:10:33.677731 20699 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:10:33.683145 20704 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:10:33.683209 20708 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:10:33.683146 20705 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:10:33.683302 20699 server_base.cc:1034] running on GCE node
I20260810 04:10:33.683631 20699 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:10:33.683889 20699 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:10:33.685070 20699 hybrid_clock.cc:648] HybridClock initialized: now 1786335033685054 us; error 52 us; skew 500 ppm
Aug 10 04:10:33 dist-test-slave-8w9v krb5kdc[20433](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335033, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:10:33.687805 20699 init.cc:377] Logged in from keytab as kudu/127.31.32.2@KRBTEST.COM (short username kudu)
I20260810 04:10:33.688256 20699 webserver.cc:460] Webserver started at http://127.31.32.2:35549/ using document root <none> and password file <none>
I20260810 04:10:33.688450 20699 fs_manager.cc:359] Metadata directory not provided
I20260810 04:10:33.688491 20699 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:10:33.688584 20699 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:10:33.689570 20699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786334794676589-31872-0/minicluster-data/ts-1/data/instance:
uuid: "8c76e4fb132743a4b6f7981a7ca3ccaa"
format_stamp: "Formatted at 2026-08-10 04:10:33 on dist-test-slave-8w9v"
server_key: "0c11c7a5a611dc24a373682419de7bba"
server_key_iv: "280c7153d3580cf82f567d7521734d55"
server_key_version: "encryptionkey@0"
I20260810 04:10:33.689878 20699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786334794676589-31872-0/minicluster-data/ts-1/wal/instance:
uuid: "8c76e4fb132743a4b6f7981a7ca3ccaa"
format_stamp: "Formatted at 2026-08-10 04:10:33 on dist-test-slave-8w9v"
server_key: "0c11c7a5a611dc24a373682419de7bba"
server_key_iv: "280c7153d3580cf82f567d7521734d55"
server_key_version: "encryptionkey@0"
I20260810 04:10:33.691195 20699 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 04:10:33.691938 20716 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:10:33.692102 20699 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 04:10:33.692219 20699 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786334794676589-31872-0/minicluster-data/ts-1/data,/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786334794676589-31872-0/minicluster-data/ts-1/wal
uuid: "8c76e4fb132743a4b6f7981a7ca3ccaa"
format_stamp: "Formatted at 2026-08-10 04:10:33 on dist-test-slave-8w9v"
server_key: "0c11c7a5a611dc24a373682419de7bba"
server_key_iv: "280c7153d3580cf82f567d7521734d55"
server_key_version: "encryptionkey@0"
I20260810 04:10:33.692276 20699 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786334794676589-31872-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786334794676589-31872-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786334794676589-31872-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:10:33.708415 20699 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:10:33.710697 20699 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:10:33.710819 20699 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:10:33.710988 20699 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:10:33.711330 20699 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:10:33.711365 20699 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:10:33.711395 20699 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:10:33.711445 20699 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:10:33.717557 20699 rpc_server.cc:307] RPC server started. Bound to: 127.31.32.2:37755
I20260810 04:10:33.717615 20846 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.32.2:37755 every 8 connection(s)
I20260810 04:10:33.717941 20699 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786334794676589-31872-0/minicluster-data/ts-1/data/info.pb
Aug 10 04:10:33 dist-test-slave-8w9v krb5kdc[20433](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335033, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.2@KRBTEST.COM for kudu/127.31.32.62@KRBTEST.COM
I20260810 04:10:33.726797 31872 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskws89Io/build/release/bin/kudu as pid 20699
I20260810 04:10:33.726919 31872 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786334794676589-31872-0/minicluster-data/ts-1/wal/instance
I20260810 04:10:33.727128 31872 external_mini_cluster.cc:1428] Setting key 263bed8f8c3bf60e8959420e33f45190
I20260810 04:10:33.727754 20849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:33.718723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:38367 (local address 127.31.32.2:55461)
0810 04:10:33.718890 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0810 04:10:33.718926 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0810 04:10:33.719458 (+ 532us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:10:33.719722 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:10:33.719725 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:10:33.720294 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:10:33.720493 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:10:33.720498 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:10:33.721245 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:10:33.721246 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:10:33.722079 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:10:33.722082 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:10:33.722483 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:10:33.722484 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:10:33.722616 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:10:33.723060 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:10:33.723068 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:10:33.724525 (+ 1457us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:10:33.726136 (+ 1611us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:10:33.726143 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:10:33.726145 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:10:33.726360 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:10:33.726541 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:10:33.726541 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:10:33.726542 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:10:33.726699 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:10:33.727110 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:10:33.727119 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:10:33.727304 (+ 185us) client_negotiation.cc:769] Sending connection context
0810 04:10:33.727424 (+ 120us) client_negotiation.cc:240] Negotiation successful
0810 04:10:33.727567 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":76,"threads_started":1}
I20260810 04:10:33.727763 20527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:33.718749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.2:55461 (local address 127.31.32.62:38367)
0810 04:10:33.718834 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0810 04:10:33.718836 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:10:33.719019 (+ 183us) server_negotiation.cc:407] Connection header received
0810 04:10:33.719576 (+ 557us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:10:33.719577 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:10:33.719589 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:10:33.719643 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:10:33.720607 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:33.721140 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:33.722181 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:33.722392 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:33.722475 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:10:33.724641 (+ 2166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:10:33.724643 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:10:33.724643 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:10:33.724651 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:10:33.726056 (+ 1405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:10:33.726428 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:10:33.726429 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:10:33.726429 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:10:33.726478 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:10:33.726778 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:10:33.726778 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:10:33.726779 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:10:33.727026 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:10:33.727116 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0810 04:10:33.727574 (+ 458us) server_negotiation.cc:299] Negotiation successful
0810 04:10:33.727662 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260810 04:10:33.728348 20847 heartbeater.cc:347] Connected to a master server at 127.31.32.62:38367
I20260810 04:10:33.728447 20847 heartbeater.cc:464] Registering TS with master...
I20260810 04:10:33.728652 20847 heartbeater.cc:511] Master 127.31.32.62:38367 requested a full tablet report, sending...
I20260810 04:10:33.729038 20483 ts_manager.cc:194] Registered new tserver with Master: 8c76e4fb132743a4b6f7981a7ca3ccaa (127.31.32.2:37755)
I20260810 04:10:33.729364 20483 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.32.2@KRBTEST.COM'} at 127.31.32.2:55461
WARNING: no policy specified for kudu/127.31.32.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.32.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.32.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.3.keytab.
Entry for principal kudu/127.31.32.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.3.keytab.
I20260810 04:10:33.755555 31872 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskws89Io/build/release/bin/kudu
/tmp/dist-test-taskws89Io/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786334794676589-31872-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786334794676589-31872-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786334794676589-31872-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786334794676589-31872-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.32.3:0
--local_ip_for_outbound_sockets=127.31.32.3
--webserver_interface=127.31.32.3
--webserver_port=0
--tserver_master_addrs=127.31.32.62:38367
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.3.keytab
--principal=kudu/127.31.32.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.32.20:39859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786334794676589-31872-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786334794676589-31872-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786334794676589-31872-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:10:33.839818 20852 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:10:33.840025 20852 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:10:33.840049 20852 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:10:33.841445 20852 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:10:33.841491 20852 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:10:33.841519 20852 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.32.3
I20260810 04:10:33.842940 20852 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.32.20:39859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786334794676589-31872-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786334794676589-31872-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.3.keytab
--principal=kudu/127.31.32.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.32.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786334794676589-31872-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.32.3
--webserver_port=0
--tserver_master_addrs=127.31.32.62:38367
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20852
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.32.3
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786334794676589-31872-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:10:33.843129 20852 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:10:33.843314 20852 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:10:33.848574 20859 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:10:33.848582 20862 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:10:33.848618 20852 server_base.cc:1034] running on GCE node
W20260810 04:10:33.848595 20860 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:10:33.849015 20852 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:10:33.849284 20852 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:10:33.850468 20852 hybrid_clock.cc:648] HybridClock initialized: now 1786335033850437 us; error 49 us; skew 500 ppm
Aug 10 04:10:33 dist-test-slave-8w9v krb5kdc[20433](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335033, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:10:33.853034 20852 init.cc:377] Logged in from keytab as kudu/127.31.32.3@KRBTEST.COM (short username kudu)
I20260810 04:10:33.853546 20852 webserver.cc:460] Webserver started at http://127.31.32.3:40241/ using document root <none> and password file <none>
I20260810 04:10:33.853788 20852 fs_manager.cc:359] Metadata directory not provided
I20260810 04:10:33.853861 20852 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:10:33.854020 20852 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:10:33.855074 20852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786334794676589-31872-0/minicluster-data/ts-2/data/instance:
uuid: "a73b02470bbc410f87b13622703b0123"
format_stamp: "Formatted at 2026-08-10 04:10:33 on dist-test-slave-8w9v"
server_key: "f7139b84a7b2fe6a307d62daaf6e6b70"
server_key_iv: "dedb8b3258cb34e4faa3c897d647ed96"
server_key_version: "encryptionkey@0"
I20260810 04:10:33.855448 20852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786334794676589-31872-0/minicluster-data/ts-2/wal/instance:
uuid: "a73b02470bbc410f87b13622703b0123"
format_stamp: "Formatted at 2026-08-10 04:10:33 on dist-test-slave-8w9v"
server_key: "f7139b84a7b2fe6a307d62daaf6e6b70"
server_key_iv: "dedb8b3258cb34e4faa3c897d647ed96"
server_key_version: "encryptionkey@0"
I20260810 04:10:33.856904 20852 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260810 04:10:33.857694 20870 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:10:33.857872 20852 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 04:10:33.857978 20852 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786334794676589-31872-0/minicluster-data/ts-2/data,/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786334794676589-31872-0/minicluster-data/ts-2/wal
uuid: "a73b02470bbc410f87b13622703b0123"
format_stamp: "Formatted at 2026-08-10 04:10:33 on dist-test-slave-8w9v"
server_key: "f7139b84a7b2fe6a307d62daaf6e6b70"
server_key_iv: "dedb8b3258cb34e4faa3c897d647ed96"
server_key_version: "encryptionkey@0"
I20260810 04:10:33.858063 20852 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786334794676589-31872-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786334794676589-31872-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786334794676589-31872-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:10:33.869412 20852 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:10:33.871367 20852 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:10:33.871549 20852 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:10:33.871793 20852 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:10:33.872198 20852 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:10:33.872265 20852 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:10:33.872331 20852 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:10:33.872380 20852 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:10:33.878758 20852 rpc_server.cc:307] RPC server started. Bound to: 127.31.32.3:36347
I20260810 04:10:33.878796 21003 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.32.3:36347 every 8 connection(s)
I20260810 04:10:33.879171 20852 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786334794676589-31872-0/minicluster-data/ts-2/data/info.pb
I20260810 04:10:33.881220 31872 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskws89Io/build/release/bin/kudu as pid 20852
I20260810 04:10:33.881311 31872 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786334794676589-31872-0/minicluster-data/ts-2/wal/instance
I20260810 04:10:33.881515 31872 external_mini_cluster.cc:1428] Setting key dd39b1ae8d98d4401a5748f08544415a
Aug 10 04:10:33 dist-test-slave-8w9v krb5kdc[20433](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335033, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.3@KRBTEST.COM for kudu/127.31.32.62@KRBTEST.COM
I20260810 04:10:33.890496 20527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:33.880006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.3:39585 (local address 127.31.32.62:38367)
0810 04:10:33.880078 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0810 04:10:33.880080 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:10:33.880339 (+ 259us) server_negotiation.cc:407] Connection header received
0810 04:10:33.880836 (+ 497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:10:33.880837 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:10:33.880847 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:10:33.880910 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:10:33.882086 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:33.882727 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:33.884765 (+ 2038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:33.884982 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:33.885066 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:10:33.887507 (+ 2441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:10:33.887509 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:10:33.887509 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:10:33.887517 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:10:33.889154 (+ 1637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:10:33.889565 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:10:33.889566 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:10:33.889566 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:10:33.889622 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:10:33.889875 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:10:33.889876 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:10:33.889876 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:10:33.889993 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:10:33.890059 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0810 04:10:33.890347 (+ 288us) server_negotiation.cc:299] Negotiation successful
0810 04:10:33.890422 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 04:10:33.890578 21006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:33.879998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:38367 (local address 127.31.32.3:39585)
0810 04:10:33.880198 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0810 04:10:33.880241 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 04:10:33.880747 (+ 506us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:10:33.880964 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:10:33.880968 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:10:33.881692 (+ 724us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:10:33.881986 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:10:33.881994 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:10:33.883786 (+ 1792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:10:33.883787 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:10:33.884668 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:10:33.884670 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:10:33.885074 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:10:33.885075 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:10:33.885214 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:10:33.885790 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:10:33.885799 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:10:33.887391 (+ 1592us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:10:33.889247 (+ 1856us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:10:33.889253 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:10:33.889255 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:10:33.889480 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:10:33.889691 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:10:33.889692 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:10:33.889692 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:10:33.889788 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:10:33.890062 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:10:33.890067 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:10:33.890176 (+ 109us) client_negotiation.cc:769] Sending connection context
0810 04:10:33.890259 (+ 83us) client_negotiation.cc:240] Negotiation successful
0810 04:10:33.890383 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":100,"threads_started":1}
I20260810 04:10:33.891100 21004 heartbeater.cc:347] Connected to a master server at 127.31.32.62:38367
I20260810 04:10:33.891218 21004 heartbeater.cc:464] Registering TS with master...
I20260810 04:10:33.891413 21004 heartbeater.cc:511] Master 127.31.32.62:38367 requested a full tablet report, sending...
I20260810 04:10:33.891882 20483 ts_manager.cc:194] Registered new tserver with Master: a73b02470bbc410f87b13622703b0123 (127.31.32.3:36347)
I20260810 04:10:33.892305 20483 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.32.3@KRBTEST.COM'} at 127.31.32.3:39585
I20260810 04:10:33.893576 31872 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 04:10:33.901032 20527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:33.895089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56578 (local address 127.31.32.62:38367)
0810 04:10:33.895144 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0810 04:10:33.895146 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:10:33.895375 (+ 229us) server_negotiation.cc:407] Connection header received
0810 04:10:33.895413 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:10:33.895413 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:10:33.895421 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:10:33.895471 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:10:33.896086 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:33.896630 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:33.897324 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:33.897500 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:33.897556 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:10:33.898787 (+ 1231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:10:33.898788 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:10:33.898788 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:10:33.898794 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:10:33.900010 (+ 1216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:10:33.900392 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:10:33.900393 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:10:33.900394 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:10:33.900446 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:10:33.900604 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:10:33.900605 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:10:33.900605 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:10:33.900748 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:10:33.900839 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0810 04:10:33.900900 (+ 61us) server_negotiation.cc:299] Negotiation successful
0810 04:10:33.900970 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260810 04:10:33.902491 20483 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56578:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 04:10:33.903970 20483 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 04:10:33.910576 21017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:33.906691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52736 (local address 127.31.32.1:39175)
0810 04:10:33.906883 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0810 04:10:33.906888 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 04:10:33.907132 (+ 244us) server_negotiation.cc:407] Connection header received
0810 04:10:33.907185 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:10:33.907187 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:10:33.907203 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:10:33.907276 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:10:33.907490 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:33.908299 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:33.909355 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:33.910128 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:33.910215 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:10:33.910285 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0810 04:10:33.910434 (+ 149us) server_negotiation.cc:299] Negotiation successful
0810 04:10:33.910487 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":102,"threads_started":1}
I20260810 04:10:33.910691 21013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:33.906442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.2:37755 (local address 127.0.0.1:32932)
0810 04:10:33.906592 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0810 04:10:33.906636 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 04:10:33.906712 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:10:33.906991 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:10:33.906995 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:10:33.906997 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:10:33.907074 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:10:33.907076 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:10:33.908143 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:10:33.908145 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:10:33.909426 (+ 1281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:10:33.909430 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:10:33.910336 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:10:33.910337 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:10:33.910422 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:10:33.910482 (+ 60us) client_negotiation.cc:769] Sending connection context
0810 04:10:33.910533 (+ 51us) client_negotiation.cc:240] Negotiation successful
0810 04:10:33.910577 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"mutex_wait_us":53,"thread_start_us":341,"threads_started":1}
I20260810 04:10:33.910599 21012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:33.906449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.1:39175 (local address 127.0.0.1:52736)
0810 04:10:33.907013 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0810 04:10:33.907023 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 04:10:33.907071 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:10:33.907360 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:10:33.907360 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:10:33.907363 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:10:33.907407 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:10:33.907409 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:10:33.908401 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:10:33.908403 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:10:33.909251 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:10:33.909253 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:10:33.910225 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:10:33.910226 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:10:33.910291 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:10:33.910317 (+ 26us) client_negotiation.cc:769] Sending connection context
0810 04:10:33.910413 (+ 96us) client_negotiation.cc:240] Negotiation successful
0810 04:10:33.910524 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"mutex_wait_us":38,"thread_start_us":56,"threads_started":1}
I20260810 04:10:33.911152 21014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:33.906621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32932 (local address 127.31.32.2:37755)
0810 04:10:33.906803 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0810 04:10:33.906808 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 04:10:33.906820 (+ 12us) server_negotiation.cc:407] Connection header received
0810 04:10:33.906862 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:10:33.906864 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:10:33.906879 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:10:33.906965 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:10:33.907171 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:33.908011 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:33.909525 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:33.910196 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:33.910258 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:10:33.910305 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0810 04:10:33.911017 (+ 712us) server_negotiation.cc:299] Negotiation successful
0810 04:10:33.911064 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":90,"threads_started":1}
I20260810 04:10:33.911563 20620 tablet_service.cc:1467] Processing CreateTablet for tablet dbb68e8aaa2c48f0802af42d909963e6 (DEFAULT_TABLE table=test-table [id=8826e10055f04e7cbfec89602f4ee573]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:10:33.911968 20620 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbb68e8aaa2c48f0802af42d909963e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:10:33.911955 20773 tablet_service.cc:1467] Processing CreateTablet for tablet dbb68e8aaa2c48f0802af42d909963e6 (DEFAULT_TABLE table=test-table [id=8826e10055f04e7cbfec89602f4ee573]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:10:33.912330 20773 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbb68e8aaa2c48f0802af42d909963e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:10:33.912700 21018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:33.906916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.3:36347 (local address 127.0.0.1:48468)
0810 04:10:33.908180 (+ 1264us) negotiation.cc:107] Waiting for socket to connect
0810 04:10:33.908190 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 04:10:33.908248 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:10:33.908495 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:10:33.908496 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:10:33.908497 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:10:33.908569 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:10:33.908570 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:10:33.910324 (+ 1754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:10:33.910326 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:10:33.911489 (+ 1163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:10:33.911492 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:10:33.912457 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:10:33.912458 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:10:33.912522 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:10:33.912526 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 04:10:33.912580 (+ 54us) client_negotiation.cc:240] Negotiation successful
0810 04:10:33.912628 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1195,"mutex_wait_us":34,"thread_start_us":54,"threads_started":1}
I20260810 04:10:33.912743 21019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:33.907013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48468 (local address 127.31.32.3:36347)
0810 04:10:33.907216 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0810 04:10:33.907220 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 04:10:33.908247 (+ 1027us) server_negotiation.cc:407] Connection header received
0810 04:10:33.908327 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:10:33.908329 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:10:33.908344 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:10:33.908419 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:10:33.908664 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:33.909730 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:33.911591 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:33.912315 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:33.912414 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:10:33.912476 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0810 04:10:33.912619 (+ 143us) server_negotiation.cc:299] Negotiation successful
0810 04:10:33.912658 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":96,"threads_started":1}
I20260810 04:10:33.913548 20928 tablet_service.cc:1467] Processing CreateTablet for tablet dbb68e8aaa2c48f0802af42d909963e6 (DEFAULT_TABLE table=test-table [id=8826e10055f04e7cbfec89602f4ee573]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:10:33.913865 20928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbb68e8aaa2c48f0802af42d909963e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:10:33.915584 21021 tablet_bootstrap.cc:492] T dbb68e8aaa2c48f0802af42d909963e6 P 02017577ae6045c58079adb7ea803efa: Bootstrap starting.
I20260810 04:10:33.915805 21020 tablet_bootstrap.cc:492] T dbb68e8aaa2c48f0802af42d909963e6 P 8c76e4fb132743a4b6f7981a7ca3ccaa: Bootstrap starting.
I20260810 04:10:33.916574 21020 tablet_bootstrap.cc:654] T dbb68e8aaa2c48f0802af42d909963e6 P 8c76e4fb132743a4b6f7981a7ca3ccaa: Neither blocks nor log segments found. Creating new log.
I20260810 04:10:33.916745 21022 tablet_bootstrap.cc:492] T dbb68e8aaa2c48f0802af42d909963e6 P a73b02470bbc410f87b13622703b0123: Bootstrap starting.
I20260810 04:10:33.916901 21021 tablet_bootstrap.cc:654] T dbb68e8aaa2c48f0802af42d909963e6 P 02017577ae6045c58079adb7ea803efa: Neither blocks nor log segments found. Creating new log.
I20260810 04:10:33.917196 21020 log.cc:824] T dbb68e8aaa2c48f0802af42d909963e6 P 8c76e4fb132743a4b6f7981a7ca3ccaa: Log is configured to *not* fsync() on all Append() calls
I20260810 04:10:33.917481 21022 tablet_bootstrap.cc:654] T dbb68e8aaa2c48f0802af42d909963e6 P a73b02470bbc410f87b13622703b0123: Neither blocks nor log segments found. Creating new log.
I20260810 04:10:33.917492 21021 log.cc:824] T dbb68e8aaa2c48f0802af42d909963e6 P 02017577ae6045c58079adb7ea803efa: Log is configured to *not* fsync() on all Append() calls
I20260810 04:10:33.918013 21020 tablet_bootstrap.cc:492] T dbb68e8aaa2c48f0802af42d909963e6 P 8c76e4fb132743a4b6f7981a7ca3ccaa: No bootstrap required, opened a new log
I20260810 04:10:33.918068 21022 log.cc:824] T dbb68e8aaa2c48f0802af42d909963e6 P a73b02470bbc410f87b13622703b0123: Log is configured to *not* fsync() on all Append() calls
I20260810 04:10:33.918114 21020 ts_tablet_manager.cc:1397] T dbb68e8aaa2c48f0802af42d909963e6 P 8c76e4fb132743a4b6f7981a7ca3ccaa: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 04:10:33.918289 21021 tablet_bootstrap.cc:492] T dbb68e8aaa2c48f0802af42d909963e6 P 02017577ae6045c58079adb7ea803efa: No bootstrap required, opened a new log
I20260810 04:10:33.918371 21021 ts_tablet_manager.cc:1397] T dbb68e8aaa2c48f0802af42d909963e6 P 02017577ae6045c58079adb7ea803efa: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 04:10:33.918814 21022 tablet_bootstrap.cc:492] T dbb68e8aaa2c48f0802af42d909963e6 P a73b02470bbc410f87b13622703b0123: No bootstrap required, opened a new log
I20260810 04:10:33.918897 21022 ts_tablet_manager.cc:1397] T dbb68e8aaa2c48f0802af42d909963e6 P a73b02470bbc410f87b13622703b0123: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 04:10:33.919931 21021 raft_consensus.cc:348] T dbb68e8aaa2c48f0802af42d909963e6 P 02017577ae6045c58079adb7ea803efa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c76e4fb132743a4b6f7981a7ca3ccaa" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 37755 } } peers { permanent_uuid: "a73b02470bbc410f87b13622703b0123" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 36347 } } peers { permanent_uuid: "02017577ae6045c58079adb7ea803efa" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 39175 } }
I20260810 04:10:33.920109 21021 raft_consensus.cc:374] T dbb68e8aaa2c48f0802af42d909963e6 P 02017577ae6045c58079adb7ea803efa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:10:33.920177 21021 raft_consensus.cc:729] T dbb68e8aaa2c48f0802af42d909963e6 P 02017577ae6045c58079adb7ea803efa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02017577ae6045c58079adb7ea803efa, State: Initialized, Role: FOLLOWER
I20260810 04:10:33.920326 21021 consensus_queue.cc:260] T dbb68e8aaa2c48f0802af42d909963e6 P 02017577ae6045c58079adb7ea803efa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c76e4fb132743a4b6f7981a7ca3ccaa" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 37755 } } peers { permanent_uuid: "a73b02470bbc410f87b13622703b0123" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 36347 } } peers { permanent_uuid: "02017577ae6045c58079adb7ea803efa" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 39175 } }
I20260810 04:10:33.920251 21020 raft_consensus.cc:348] T dbb68e8aaa2c48f0802af42d909963e6 P 8c76e4fb132743a4b6f7981a7ca3ccaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c76e4fb132743a4b6f7981a7ca3ccaa" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 37755 } } peers { permanent_uuid: "a73b02470bbc410f87b13622703b0123" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 36347 } } peers { permanent_uuid: "02017577ae6045c58079adb7ea803efa" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 39175 } }
I20260810 04:10:33.920433 21020 raft_consensus.cc:374] T dbb68e8aaa2c48f0802af42d909963e6 P 8c76e4fb132743a4b6f7981a7ca3ccaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:10:33.920477 21020 raft_consensus.cc:729] T dbb68e8aaa2c48f0802af42d909963e6 P 8c76e4fb132743a4b6f7981a7ca3ccaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c76e4fb132743a4b6f7981a7ca3ccaa, State: Initialized, Role: FOLLOWER
I20260810 04:10:33.920635 21021 ts_tablet_manager.cc:1428] T dbb68e8aaa2c48f0802af42d909963e6 P 02017577ae6045c58079adb7ea803efa: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260810 04:10:33.920608 21020 consensus_queue.cc:260] T dbb68e8aaa2c48f0802af42d909963e6 P 8c76e4fb132743a4b6f7981a7ca3ccaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c76e4fb132743a4b6f7981a7ca3ccaa" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 37755 } } peers { permanent_uuid: "a73b02470bbc410f87b13622703b0123" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 36347 } } peers { permanent_uuid: "02017577ae6045c58079adb7ea803efa" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 39175 } }
I20260810 04:10:33.920799 20694 heartbeater.cc:503] Master 127.31.32.62:38367 was elected leader, sending a full tablet report...
I20260810 04:10:33.920893 21020 ts_tablet_manager.cc:1428] T dbb68e8aaa2c48f0802af42d909963e6 P 8c76e4fb132743a4b6f7981a7ca3ccaa: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 04:10:33.921031 20847 heartbeater.cc:503] Master 127.31.32.62:38367 was elected leader, sending a full tablet report...
I20260810 04:10:33.920918 21022 raft_consensus.cc:348] T dbb68e8aaa2c48f0802af42d909963e6 P a73b02470bbc410f87b13622703b0123 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c76e4fb132743a4b6f7981a7ca3ccaa" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 37755 } } peers { permanent_uuid: "a73b02470bbc410f87b13622703b0123" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 36347 } } peers { permanent_uuid: "02017577ae6045c58079adb7ea803efa" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 39175 } }
I20260810 04:10:33.921087 21022 raft_consensus.cc:374] T dbb68e8aaa2c48f0802af42d909963e6 P a73b02470bbc410f87b13622703b0123 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:10:33.921123 21022 raft_consensus.cc:729] T dbb68e8aaa2c48f0802af42d909963e6 P a73b02470bbc410f87b13622703b0123 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a73b02470bbc410f87b13622703b0123, State: Initialized, Role: FOLLOWER
I20260810 04:10:33.921226 21022 consensus_queue.cc:260] T dbb68e8aaa2c48f0802af42d909963e6 P a73b02470bbc410f87b13622703b0123 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c76e4fb132743a4b6f7981a7ca3ccaa" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 37755 } } peers { permanent_uuid: "a73b02470bbc410f87b13622703b0123" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 36347 } } peers { permanent_uuid: "02017577ae6045c58079adb7ea803efa" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 39175 } }
I20260810 04:10:33.921482 21022 ts_tablet_manager.cc:1428] T dbb68e8aaa2c48f0802af42d909963e6 P a73b02470bbc410f87b13622703b0123: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 04:10:33.921506 21004 heartbeater.cc:503] Master 127.31.32.62:38367 was elected leader, sending a full tablet report...
W20260810 04:10:33.968588 20848 tablet.cc:2369] T dbb68e8aaa2c48f0802af42d909963e6 P 8c76e4fb132743a4b6f7981a7ca3ccaa: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 04:10:34.059664 20695 tablet.cc:2369] T dbb68e8aaa2c48f0802af42d909963e6 P 02017577ae6045c58079adb7ea803efa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 04:10:34.068356 21028 raft_consensus.cc:482] T dbb68e8aaa2c48f0802af42d909963e6 P 8c76e4fb132743a4b6f7981a7ca3ccaa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 04:10:34.068488 21028 raft_consensus.cc:504] T dbb68e8aaa2c48f0802af42d909963e6 P 8c76e4fb132743a4b6f7981a7ca3ccaa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c76e4fb132743a4b6f7981a7ca3ccaa" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 37755 } } peers { permanent_uuid: "a73b02470bbc410f87b13622703b0123" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 36347 } } peers { permanent_uuid: "02017577ae6045c58079adb7ea803efa" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 39175 } }
I20260810 04:10:34.068971 21028 leader_election.cc:288] T dbb68e8aaa2c48f0802af42d909963e6 P 8c76e4fb132743a4b6f7981a7ca3ccaa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a73b02470bbc410f87b13622703b0123 (127.31.32.3:36347), 02017577ae6045c58079adb7ea803efa (127.31.32.1:39175)
I20260810 04:10:34.072225 21019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:34.069270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.2:42301 (local address 127.31.32.3:36347)
0810 04:10:34.069316 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0810 04:10:34.069319 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:10:34.069330 (+ 11us) server_negotiation.cc:407] Connection header received
0810 04:10:34.069396 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:10:34.069397 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:10:34.069410 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:10:34.069487 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:10:34.069662 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:34.070458 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:34.071404 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:34.071800 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:34.071892 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:10:34.071905 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0810 04:10:34.072094 (+ 189us) server_negotiation.cc:299] Negotiation successful
0810 04:10:34.072165 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260810 04:10:34.072223 20849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:34.069187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.3:36347 (local address 127.31.32.2:42301)
0810 04:10:34.069241 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0810 04:10:34.069254 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 04:10:34.069335 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:10:34.069508 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:10:34.069509 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:10:34.069511 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:10:34.069585 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:10:34.069587 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:10:34.070544 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:10:34.070545 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:10:34.071290 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:10:34.071292 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:10:34.071893 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:10:34.071895 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:10:34.071975 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:10:34.071978 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 04:10:34.072077 (+ 99us) client_negotiation.cc:240] Negotiation successful
0810 04:10:34.072146 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260810 04:10:34.072307 21017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:34.069289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.2:55805 (local address 127.31.32.1:39175)
0810 04:10:34.069347 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0810 04:10:34.069350 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:10:34.069535 (+ 185us) server_negotiation.cc:407] Connection header received
0810 04:10:34.069637 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:10:34.069638 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:10:34.069656 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:10:34.069716 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:10:34.069892 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:34.070570 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:34.071515 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:34.071889 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:34.071978 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:10:34.071990 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0810 04:10:34.072188 (+ 198us) server_negotiation.cc:299] Negotiation successful
0810 04:10:34.072234 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260810 04:10:34.072348 21031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:34.069196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.1:39175 (local address 127.31.32.2:55805)
0810 04:10:34.069439 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0810 04:10:34.069449 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 04:10:34.069559 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:10:34.069741 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:10:34.069742 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:10:34.069743 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:10:34.069807 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:10:34.069808 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:10:34.070667 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:10:34.070668 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:10:34.071410 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:10:34.071412 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:10:34.071981 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:10:34.071982 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:10:34.072072 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:10:34.072075 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 04:10:34.072179 (+ 104us) client_negotiation.cc:240] Negotiation successful
0810 04:10:34.072220 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"mutex_wait_us":57,"thread_start_us":131,"threads_started":1}
I20260810 04:10:34.072642 20952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbb68e8aaa2c48f0802af42d909963e6" candidate_uuid: "8c76e4fb132743a4b6f7981a7ca3ccaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a73b02470bbc410f87b13622703b0123" is_pre_election: true
I20260810 04:10:34.072655 20641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbb68e8aaa2c48f0802af42d909963e6" candidate_uuid: "8c76e4fb132743a4b6f7981a7ca3ccaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02017577ae6045c58079adb7ea803efa" is_pre_election: true
I20260810 04:10:34.072825 20641 raft_consensus.cc:2454] T dbb68e8aaa2c48f0802af42d909963e6 P 02017577ae6045c58079adb7ea803efa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c76e4fb132743a4b6f7981a7ca3ccaa in term 0.
I20260810 04:10:34.072824 20952 raft_consensus.cc:2454] T dbb68e8aaa2c48f0802af42d909963e6 P a73b02470bbc410f87b13622703b0123 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c76e4fb132743a4b6f7981a7ca3ccaa in term 0.
I20260810 04:10:34.073050 20720 leader_election.cc:302] T dbb68e8aaa2c48f0802af42d909963e6 P 8c76e4fb132743a4b6f7981a7ca3ccaa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02017577ae6045c58079adb7ea803efa, 8c76e4fb132743a4b6f7981a7ca3ccaa; no voters:
I20260810 04:10:34.073192 21028 raft_consensus.cc:2781] T dbb68e8aaa2c48f0802af42d909963e6 P 8c76e4fb132743a4b6f7981a7ca3ccaa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 04:10:34.073251 21028 raft_consensus.cc:482] T dbb68e8aaa2c48f0802af42d909963e6 P 8c76e4fb132743a4b6f7981a7ca3ccaa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 04:10:34.073273 21028 raft_consensus.cc:3037] T dbb68e8aaa2c48f0802af42d909963e6 P 8c76e4fb132743a4b6f7981a7ca3ccaa [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:10:34.073999 21028 raft_consensus.cc:504] T dbb68e8aaa2c48f0802af42d909963e6 P 8c76e4fb132743a4b6f7981a7ca3ccaa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c76e4fb132743a4b6f7981a7ca3ccaa" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 37755 } } peers { permanent_uuid: "a73b02470bbc410f87b13622703b0123" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 36347 } } peers { permanent_uuid: "02017577ae6045c58079adb7ea803efa" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 39175 } }
I20260810 04:10:34.074154 21028 leader_election.cc:288] T dbb68e8aaa2c48f0802af42d909963e6 P 8c76e4fb132743a4b6f7981a7ca3ccaa [CANDIDATE]: Term 1 election: Requested vote from peers a73b02470bbc410f87b13622703b0123 (127.31.32.3:36347), 02017577ae6045c58079adb7ea803efa (127.31.32.1:39175)
I20260810 04:10:34.074333 20952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbb68e8aaa2c48f0802af42d909963e6" candidate_uuid: "8c76e4fb132743a4b6f7981a7ca3ccaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a73b02470bbc410f87b13622703b0123"
I20260810 04:10:34.074352 20641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbb68e8aaa2c48f0802af42d909963e6" candidate_uuid: "8c76e4fb132743a4b6f7981a7ca3ccaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02017577ae6045c58079adb7ea803efa"
I20260810 04:10:34.074435 20952 raft_consensus.cc:3037] T dbb68e8aaa2c48f0802af42d909963e6 P a73b02470bbc410f87b13622703b0123 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:10:34.074435 20641 raft_consensus.cc:3037] T dbb68e8aaa2c48f0802af42d909963e6 P 02017577ae6045c58079adb7ea803efa [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:10:34.075366 20641 raft_consensus.cc:2454] T dbb68e8aaa2c48f0802af42d909963e6 P 02017577ae6045c58079adb7ea803efa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c76e4fb132743a4b6f7981a7ca3ccaa in term 1.
I20260810 04:10:34.075366 20952 raft_consensus.cc:2454] T dbb68e8aaa2c48f0802af42d909963e6 P a73b02470bbc410f87b13622703b0123 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c76e4fb132743a4b6f7981a7ca3ccaa in term 1.
I20260810 04:10:34.075546 20720 leader_election.cc:302] T dbb68e8aaa2c48f0802af42d909963e6 P 8c76e4fb132743a4b6f7981a7ca3ccaa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02017577ae6045c58079adb7ea803efa, 8c76e4fb132743a4b6f7981a7ca3ccaa; no voters:
I20260810 04:10:34.075670 21028 raft_consensus.cc:2781] T dbb68e8aaa2c48f0802af42d909963e6 P 8c76e4fb132743a4b6f7981a7ca3ccaa [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:10:34.075858 21028 raft_consensus.cc:686] T dbb68e8aaa2c48f0802af42d909963e6 P 8c76e4fb132743a4b6f7981a7ca3ccaa [term 1 LEADER]: Becoming Leader. State: Replica: 8c76e4fb132743a4b6f7981a7ca3ccaa, State: Running, Role: LEADER
I20260810 04:10:34.075968 21028 consensus_queue.cc:237] T dbb68e8aaa2c48f0802af42d909963e6 P 8c76e4fb132743a4b6f7981a7ca3ccaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c76e4fb132743a4b6f7981a7ca3ccaa" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 37755 } } peers { permanent_uuid: "a73b02470bbc410f87b13622703b0123" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 36347 } } peers { permanent_uuid: "02017577ae6045c58079adb7ea803efa" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 39175 } }
I20260810 04:10:34.076987 20482 catalog_manager.cc:5495] T dbb68e8aaa2c48f0802af42d909963e6 P 8c76e4fb132743a4b6f7981a7ca3ccaa reported cstate change: term changed from 0 to 1, leader changed from <none> to 8c76e4fb132743a4b6f7981a7ca3ccaa (127.31.32.2). New cstate: current_term: 1 leader_uuid: "8c76e4fb132743a4b6f7981a7ca3ccaa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c76e4fb132743a4b6f7981a7ca3ccaa" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 37755 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a73b02470bbc410f87b13622703b0123" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 36347 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "02017577ae6045c58079adb7ea803efa" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 39175 } health_report { overall_health: UNKNOWN } } }
I20260810 04:10:34.096649 21014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:34.093914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32944 (local address 127.31.32.2:37755)
0810 04:10:34.093980 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0810 04:10:34.093983 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:10:34.093993 (+ 10us) server_negotiation.cc:407] Connection header received
0810 04:10:34.094081 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:10:34.094082 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:10:34.094119 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:10:34.094203 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 04:10:34.094368 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:34.094869 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:34.095759 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:34.095958 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:34.096078 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:10:34.096289 (+ 211us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 04:10:34.096403 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 04:10:34.096509 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0810 04:10:34.096529 (+ 20us) server_negotiation.cc:299] Negotiation successful
0810 04:10:34.096550 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260810 04:10:34.098559 20641 raft_consensus.cc:1261] T dbb68e8aaa2c48f0802af42d909963e6 P 02017577ae6045c58079adb7ea803efa [term 1 FOLLOWER]: Refusing update from remote peer 8c76e4fb132743a4b6f7981a7ca3ccaa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:10:34.098558 20952 raft_consensus.cc:1261] T dbb68e8aaa2c48f0802af42d909963e6 P a73b02470bbc410f87b13622703b0123 [term 1 FOLLOWER]: Refusing update from remote peer 8c76e4fb132743a4b6f7981a7ca3ccaa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:10:34.099007 21028 consensus_queue.cc:1035] T dbb68e8aaa2c48f0802af42d909963e6 P 8c76e4fb132743a4b6f7981a7ca3ccaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "02017577ae6045c58079adb7ea803efa" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 39175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:10:34.099174 21032 consensus_queue.cc:1035] T dbb68e8aaa2c48f0802af42d909963e6 P 8c76e4fb132743a4b6f7981a7ca3ccaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "a73b02470bbc410f87b13622703b0123" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 36347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:10:34.102054 21036 mvcc.cc:205] Tried to move back new op lower bound from 7316828299664695296 to 7316828299575939072. Current Snapshot: MvccSnapshot[applied={T|T < 7316828299664695296}]
I20260810 04:10:34.104080 20481 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56578:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 04:10:34.104233 20481 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56578:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 04:10:34.105063 20481 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 19f71e13a69a4503b21addf1c8a02317: Sending DeleteTablet for 3 replicas of tablet dbb68e8aaa2c48f0802af42d909963e6
I20260810 04:10:34.105520 20773 tablet_service.cc:1514] Processing DeleteTablet for tablet dbb68e8aaa2c48f0802af42d909963e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:10:34 UTC) from {username='kudu', principal='kudu/127.31.32.62@KRBTEST.COM'} at 127.0.0.1:32932
I20260810 04:10:34.105576 20620 tablet_service.cc:1514] Processing DeleteTablet for tablet dbb68e8aaa2c48f0802af42d909963e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:10:34 UTC) from {username='kudu', principal='kudu/127.31.32.62@KRBTEST.COM'} at 127.0.0.1:52736
I20260810 04:10:34.105669 20928 tablet_service.cc:1514] Processing DeleteTablet for tablet dbb68e8aaa2c48f0802af42d909963e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:10:34 UTC) from {username='kudu', principal='kudu/127.31.32.62@KRBTEST.COM'} at 127.0.0.1:48468
I20260810 04:10:34.105937 21047 tablet_replica.cc:330] T dbb68e8aaa2c48f0802af42d909963e6 P a73b02470bbc410f87b13622703b0123: stopping tablet replica
I20260810 04:10:34.106035 21047 raft_consensus.cc:2229] T dbb68e8aaa2c48f0802af42d909963e6 P a73b02470bbc410f87b13622703b0123 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 04:10:34.106310 21046 tablet_replica.cc:330] T dbb68e8aaa2c48f0802af42d909963e6 P 02017577ae6045c58079adb7ea803efa: stopping tablet replica
I20260810 04:10:34.106391 21045 tablet_replica.cc:330] T dbb68e8aaa2c48f0802af42d909963e6 P 8c76e4fb132743a4b6f7981a7ca3ccaa: stopping tablet replica
I20260810 04:10:34.106410 21046 raft_consensus.cc:2229] T dbb68e8aaa2c48f0802af42d909963e6 P 02017577ae6045c58079adb7ea803efa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 04:10:34.106492 21045 raft_consensus.cc:2229] T dbb68e8aaa2c48f0802af42d909963e6 P 8c76e4fb132743a4b6f7981a7ca3ccaa [term 1 LEADER]: Raft consensus shutting down.
I20260810 04:10:34.106504 21046 raft_consensus.cc:2258] T dbb68e8aaa2c48f0802af42d909963e6 P 02017577ae6045c58079adb7ea803efa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:10:34.106158 21047 raft_consensus.cc:2258] T dbb68e8aaa2c48f0802af42d909963e6 P a73b02470bbc410f87b13622703b0123 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:10:34.106608 31872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskws89Io/build/release/bin/kudu with pid 20551
I20260810 04:10:34.111611 21045 raft_consensus.cc:2258] T dbb68e8aaa2c48f0802af42d909963e6 P 8c76e4fb132743a4b6f7981a7ca3ccaa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:10:34.112453 21045 ts_tablet_manager.cc:1905] T dbb68e8aaa2c48f0802af42d909963e6 P 8c76e4fb132743a4b6f7981a7ca3ccaa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:10:34.113690 21047 ts_tablet_manager.cc:1905] T dbb68e8aaa2c48f0802af42d909963e6 P a73b02470bbc410f87b13622703b0123: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:10:34.114832 21045 ts_tablet_manager.cc:1918] T dbb68e8aaa2c48f0802af42d909963e6 P 8c76e4fb132743a4b6f7981a7ca3ccaa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 04:10:34.114907 21045 log.cc:1196] T dbb68e8aaa2c48f0802af42d909963e6 P 8c76e4fb132743a4b6f7981a7ca3ccaa: Deleting WAL directory at /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786334794676589-31872-0/minicluster-data/ts-1/wal/wals/dbb68e8aaa2c48f0802af42d909963e6
I20260810 04:10:34.115197 21045 ts_tablet_manager.cc:1939] T dbb68e8aaa2c48f0802af42d909963e6 P 8c76e4fb132743a4b6f7981a7ca3ccaa: Deleting consensus metadata
I20260810 04:10:34.115648 20468 catalog_manager.cc:4841] TS 8c76e4fb132743a4b6f7981a7ca3ccaa (127.31.32.2:37755): tablet dbb68e8aaa2c48f0802af42d909963e6 (table test-table [id=8826e10055f04e7cbfec89602f4ee573]) successfully deleted
I20260810 04:10:34.115720 21047 ts_tablet_manager.cc:1918] T dbb68e8aaa2c48f0802af42d909963e6 P a73b02470bbc410f87b13622703b0123: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 04:10:34.115799 21047 log.cc:1196] T dbb68e8aaa2c48f0802af42d909963e6 P a73b02470bbc410f87b13622703b0123: Deleting WAL directory at /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786334794676589-31872-0/minicluster-data/ts-2/wal/wals/dbb68e8aaa2c48f0802af42d909963e6
I20260810 04:10:34.116114 21047 ts_tablet_manager.cc:1939] T dbb68e8aaa2c48f0802af42d909963e6 P a73b02470bbc410f87b13622703b0123: Deleting consensus metadata
I20260810 04:10:34.116585 20468 catalog_manager.cc:4841] TS a73b02470bbc410f87b13622703b0123 (127.31.32.3:36347): tablet dbb68e8aaa2c48f0802af42d909963e6 (table test-table [id=8826e10055f04e7cbfec89602f4ee573]) successfully deleted
W20260810 04:10:34.118826 20470 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.32.1:39175 (error 108)
I20260810 04:10:34.119097 31872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskws89Io/build/release/bin/kudu with pid 20699
I20260810 04:10:34.120252 21018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:34.119281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.1:39175 (local address 127.0.0.1:52738)
0810 04:10:34.120159 (+ 878us) negotiation.cc:107] Waiting for socket to connect
0810 04:10:34.120209 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.1:39175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":853}
W20260810 04:10:34.121688 20470 catalog_manager.cc:4584] TS 02017577ae6045c58079adb7ea803efa (127.31.32.1:39175): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet dbb68e8aaa2c48f0802af42d909963e6: Network error: Client connection negotiation failed: client connection to 127.31.32.1:39175: connect: Connection refused (error 111)
I20260810 04:10:34.125198 31872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskws89Io/build/release/bin/kudu with pid 20852
I20260810 04:10:34.132727 31872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskws89Io/build/release/bin/kudu with pid 20447
2026-08-10T04:10:34Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2142 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-08-10T04:10:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T04:10:34Z Disabled control of system clock
I20260810 04:10:34.163957 31872 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskws89Io/build/release/bin/kudu
/tmp/dist-test-taskws89Io/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786334794676589-31872-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786334794676589-31872-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786334794676589-31872-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786334794676589-31872-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.32.62:36631
--webserver_interface=127.31.32.62
--webserver_port=0
--builtin_ntp_servers=127.31.32.20:40207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.32.62:36631
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 04:10:34.247866 21052 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:10:34.248181 21052 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:10:34.248210 21052 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:10:34.249296 21052 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:10:34.249341 21052 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:10:34.249353 21052 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:10:34.249366 21052 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:10:34.249378 21052 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:10:34.250659 21052 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.32.20:40207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786334794676589-31872-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786334794676589-31872-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.32.62:36631
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.32.62:36631
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786334794676589-31872-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.32.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21052
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786334794676589-31872-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:10:34.250844 21052 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:10:34.251087 21052 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:10:34.256174 21062 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:10:34.256218 21058 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:10:34.256207 21059 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:10:34.256387 21052 server_base.cc:1034] running on GCE node
I20260810 04:10:34.256635 21052 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:10:34.256964 21052 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:10:34.258133 21052 hybrid_clock.cc:648] HybridClock initialized: now 1786335034258108 us; error 43 us; skew 500 ppm
I20260810 04:10:34.259483 21052 webserver.cc:460] Webserver started at http://127.31.32.62:40575/ using document root <none> and password file <none>
I20260810 04:10:34.259778 21052 fs_manager.cc:359] Metadata directory not provided
I20260810 04:10:34.259871 21052 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:10:34.260043 21052 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:10:34.261219 21052 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786334794676589-31872-0/minicluster-data/master-0/data/instance:
uuid: "4a49c1b253d746aeb3699de0c1505ce6"
format_stamp: "Formatted at 2026-08-10 04:10:34 on dist-test-slave-8w9v"
server_key: "4144c54d4331351682de03af950e378b"
server_key_iv: "87a81a90fcbdf71a63eac6921fa93fd4"
server_key_version: "encryptionkey@0"
I20260810 04:10:34.261606 21052 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786334794676589-31872-0/minicluster-data/master-0/wal/instance:
uuid: "4a49c1b253d746aeb3699de0c1505ce6"
format_stamp: "Formatted at 2026-08-10 04:10:34 on dist-test-slave-8w9v"
server_key: "4144c54d4331351682de03af950e378b"
server_key_iv: "87a81a90fcbdf71a63eac6921fa93fd4"
server_key_version: "encryptionkey@0"
I20260810 04:10:34.263001 21052 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 04:10:34.263793 21068 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:10:34.264173 21052 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:10:34.264324 21052 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786334794676589-31872-0/minicluster-data/master-0/data,/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786334794676589-31872-0/minicluster-data/master-0/wal
uuid: "4a49c1b253d746aeb3699de0c1505ce6"
format_stamp: "Formatted at 2026-08-10 04:10:34 on dist-test-slave-8w9v"
server_key: "4144c54d4331351682de03af950e378b"
server_key_iv: "87a81a90fcbdf71a63eac6921fa93fd4"
server_key_version: "encryptionkey@0"
I20260810 04:10:34.264413 21052 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786334794676589-31872-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786334794676589-31872-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786334794676589-31872-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:10:34.276033 21052 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:10:34.276439 21052 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:10:34.276610 21052 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:10:34.281039 21052 rpc_server.cc:307] RPC server started. Bound to: 127.31.32.62:36631
I20260810 04:10:34.281082 21131 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.32.62:36631 every 8 connection(s)
I20260810 04:10:34.281448 21052 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786334794676589-31872-0/minicluster-data/master-0/data/info.pb
I20260810 04:10:34.282455 21133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:10:34.284893 21133 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a49c1b253d746aeb3699de0c1505ce6: Bootstrap starting.
I20260810 04:10:34.285458 21133 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4a49c1b253d746aeb3699de0c1505ce6: Neither blocks nor log segments found. Creating new log.
I20260810 04:10:34.285702 21133 log.cc:824] T 00000000000000000000000000000000 P 4a49c1b253d746aeb3699de0c1505ce6: Log is configured to *not* fsync() on all Append() calls
I20260810 04:10:34.286389 21133 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a49c1b253d746aeb3699de0c1505ce6: No bootstrap required, opened a new log
I20260810 04:10:34.287892 21133 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4a49c1b253d746aeb3699de0c1505ce6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a49c1b253d746aeb3699de0c1505ce6" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 36631 } }
I20260810 04:10:34.288030 21133 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4a49c1b253d746aeb3699de0c1505ce6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:10:34.288053 21133 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4a49c1b253d746aeb3699de0c1505ce6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a49c1b253d746aeb3699de0c1505ce6, State: Initialized, Role: FOLLOWER
I20260810 04:10:34.288252 21133 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4a49c1b253d746aeb3699de0c1505ce6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a49c1b253d746aeb3699de0c1505ce6" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 36631 } }
I20260810 04:10:34.288349 21133 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4a49c1b253d746aeb3699de0c1505ce6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 04:10:34.288375 21133 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4a49c1b253d746aeb3699de0c1505ce6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 04:10:34.288405 21133 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4a49c1b253d746aeb3699de0c1505ce6 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:10:34.288928 21133 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4a49c1b253d746aeb3699de0c1505ce6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a49c1b253d746aeb3699de0c1505ce6" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 36631 } }
I20260810 04:10:34.289000 31872 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskws89Io/build/release/bin/kudu as pid 21052
I20260810 04:10:34.289048 21133 leader_election.cc:302] T 00000000000000000000000000000000 P 4a49c1b253d746aeb3699de0c1505ce6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4a49c1b253d746aeb3699de0c1505ce6; no voters:
I20260810 04:10:34.289106 31872 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786334794676589-31872-0/minicluster-data/master-0/wal/instance
I20260810 04:10:34.289230 21133 leader_election.cc:288] T 00000000000000000000000000000000 P 4a49c1b253d746aeb3699de0c1505ce6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 04:10:34.289327 31872 external_mini_cluster.cc:1428] Setting key 6b6eef67691b1f3ca8f42985bf241da1
I20260810 04:10:34.289431 21137 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4a49c1b253d746aeb3699de0c1505ce6 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:10:34.289521 21133 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4a49c1b253d746aeb3699de0c1505ce6 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:10:34.289625 21137 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4a49c1b253d746aeb3699de0c1505ce6 [term 1 LEADER]: Becoming Leader. State: Replica: 4a49c1b253d746aeb3699de0c1505ce6, State: Running, Role: LEADER
I20260810 04:10:34.289777 21137 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4a49c1b253d746aeb3699de0c1505ce6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a49c1b253d746aeb3699de0c1505ce6" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 36631 } }
I20260810 04:10:34.290189 21137 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a49c1b253d746aeb3699de0c1505ce6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4a49c1b253d746aeb3699de0c1505ce6. Latest consensus state: current_term: 1 leader_uuid: "4a49c1b253d746aeb3699de0c1505ce6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a49c1b253d746aeb3699de0c1505ce6" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 36631 } } }
I20260810 04:10:34.290289 21137 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a49c1b253d746aeb3699de0c1505ce6 [sys.catalog]: This master's current role is: LEADER
I20260810 04:10:34.290419 21139 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a49c1b253d746aeb3699de0c1505ce6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4a49c1b253d746aeb3699de0c1505ce6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a49c1b253d746aeb3699de0c1505ce6" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 36631 } } }
I20260810 04:10:34.290534 21150 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 04:10:34.290552 21139 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a49c1b253d746aeb3699de0c1505ce6 [sys.catalog]: This master's current role is: LEADER
I20260810 04:10:34.291123 21150 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 04:10:34.292984 21150 catalog_manager.cc:1347] Generated new cluster ID: e6a218ea00524d28b265ca0928ef5202
I20260810 04:10:34.293040 21150 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 04:10:34.296077 21146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:34.289989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51590 (local address 127.31.32.62:36631)
0810 04:10:34.290369 (+ 380us) server_negotiation.cc:206] Beginning negotiation
0810 04:10:34.290380 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 04:10:34.290951 (+ 571us) server_negotiation.cc:407] Connection header received
0810 04:10:34.291089 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:10:34.291096 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:10:34.291175 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:10:34.291332 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:10:34.291612 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:34.292981 (+ 1369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:34.293847 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:34.294300 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:34.294452 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:10:34.295259 (+ 807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:10:34.295263 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:10:34.295264 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:10:34.295456 (+ 192us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:10:34.295494 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:10:34.295505 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:10:34.295598 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0810 04:10:34.295700 (+ 102us) server_negotiation.cc:299] Negotiation successful
0810 04:10:34.295889 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":84,"threads_started":1}
I20260810 04:10:34.307232 21150 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 04:10:34.307895 21150 catalog_manager.cc:1504] Loading token signing keys...
I20260810 04:10:34.313272 21150 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4a49c1b253d746aeb3699de0c1505ce6: Generated new TSK 0
I20260810 04:10:34.313483 21150 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 04:10:34.319103 31872 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskws89Io/build/release/bin/kudu
/tmp/dist-test-taskws89Io/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786334794676589-31872-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786334794676589-31872-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786334794676589-31872-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786334794676589-31872-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.32.1:0
--local_ip_for_outbound_sockets=127.31.32.1
--webserver_interface=127.31.32.1
--webserver_port=0
--tserver_master_addrs=127.31.32.62:36631
--builtin_ntp_servers=127.31.32.20:40207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 04:10:34.403766 21161 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:10:34.404057 21161 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:10:34.404096 21161 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:10:34.405359 21161 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:10:34.405437 21161 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:10:34.405483 21161 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.32.1
I20260810 04:10:34.407071 21161 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.32.20:40207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786334794676589-31872-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786334794676589-31872-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.32.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786334794676589-31872-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.32.1
--webserver_port=0
--tserver_master_addrs=127.31.32.62:36631
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21161
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.32.1
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786334794676589-31872-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:10:34.407361 21161 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:10:34.407728 21161 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:10:34.414175 21169 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:10:34.414175 21167 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:10:34.414184 21166 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:10:34.414413 21161 server_base.cc:1034] running on GCE node
I20260810 04:10:34.414870 21161 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:10:34.415186 21161 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:10:34.416370 21161 hybrid_clock.cc:648] HybridClock initialized: now 1786335034416358 us; error 31 us; skew 500 ppm
I20260810 04:10:34.417785 21161 webserver.cc:460] Webserver started at http://127.31.32.1:39133/ using document root <none> and password file <none>
I20260810 04:10:34.418066 21161 fs_manager.cc:359] Metadata directory not provided
I20260810 04:10:34.418123 21161 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:10:34.418246 21161 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:10:34.419658 21161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786334794676589-31872-0/minicluster-data/ts-0/data/instance:
uuid: "a64e52bddcbe48bf84196986142ba744"
format_stamp: "Formatted at 2026-08-10 04:10:34 on dist-test-slave-8w9v"
server_key: "df7c3c47d45d2e093643ac4920488eb3"
server_key_iv: "fa45ebecfb1b02c12f23f0df9b4d97ef"
server_key_version: "encryptionkey@0"
I20260810 04:10:34.420051 21161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786334794676589-31872-0/minicluster-data/ts-0/wal/instance:
uuid: "a64e52bddcbe48bf84196986142ba744"
format_stamp: "Formatted at 2026-08-10 04:10:34 on dist-test-slave-8w9v"
server_key: "df7c3c47d45d2e093643ac4920488eb3"
server_key_iv: "fa45ebecfb1b02c12f23f0df9b4d97ef"
server_key_version: "encryptionkey@0"
I20260810 04:10:34.421831 21161 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.001s
I20260810 04:10:34.422816 21176 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:10:34.423044 21161 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 04:10:34.423138 21161 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786334794676589-31872-0/minicluster-data/ts-0/data,/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786334794676589-31872-0/minicluster-data/ts-0/wal
uuid: "a64e52bddcbe48bf84196986142ba744"
format_stamp: "Formatted at 2026-08-10 04:10:34 on dist-test-slave-8w9v"
server_key: "df7c3c47d45d2e093643ac4920488eb3"
server_key_iv: "fa45ebecfb1b02c12f23f0df9b4d97ef"
server_key_version: "encryptionkey@0"
I20260810 04:10:34.423213 21161 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786334794676589-31872-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786334794676589-31872-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786334794676589-31872-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:10:34.442869 21161 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:10:34.443173 21161 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:10:34.443284 21161 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:10:34.443502 21161 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:10:34.443796 21161 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:10:34.443828 21161 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:10:34.443853 21161 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:10:34.443866 21161 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:10:34.449864 21161 rpc_server.cc:307] RPC server started. Bound to: 127.31.32.1:33441
I20260810 04:10:34.449913 21304 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.32.1:33441 every 8 connection(s)
I20260810 04:10:34.450274 21161 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786334794676589-31872-0/minicluster-data/ts-0/data/info.pb
I20260810 04:10:34.455267 31872 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskws89Io/build/release/bin/kudu as pid 21161
I20260810 04:10:34.455427 31872 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786334794676589-31872-0/minicluster-data/ts-0/wal/instance
I20260810 04:10:34.455586 21146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:34.451093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.1:42319 (local address 127.31.32.62:36631)
0810 04:10:34.451170 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0810 04:10:34.451173 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:10:34.451442 (+ 269us) server_negotiation.cc:407] Connection header received
0810 04:10:34.451915 (+ 473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:10:34.451916 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:10:34.451928 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:10:34.451983 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:10:34.452741 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:34.453244 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:34.454187 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:34.454368 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:34.454419 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:10:34.455302 (+ 883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:10:34.455303 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:10:34.455304 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:10:34.455308 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:10:34.455324 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:10:34.455329 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:10:34.455389 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0810 04:10:34.455467 (+ 78us) server_negotiation.cc:299] Negotiation successful
0810 04:10:34.455504 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260810 04:10:34.455657 31872 external_mini_cluster.cc:1428] Setting key f556166dfe7704231c6986630a62a499
I20260810 04:10:34.455883 21307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:34.451088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:36631 (local address 127.31.32.1:42319)
0810 04:10:34.451307 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0810 04:10:34.451344 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0810 04:10:34.451826 (+ 482us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:10:34.452037 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:10:34.452040 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:10:34.452048 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:10:34.452658 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:10:34.452664 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:10:34.453330 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:10:34.453331 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:10:34.454108 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:10:34.454110 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:10:34.454448 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:10:34.454449 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:10:34.454562 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:10:34.455134 (+ 572us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 04:10:34.455135 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:10:34.455176 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 04:10:34.455208 (+ 32us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:10:34.455395 (+ 187us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:10:34.455400 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:10:34.455403 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 04:10:34.455472 (+ 69us) client_negotiation.cc:240] Negotiation successful
0810 04:10:34.455575 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":113,"threads_started":1}
I20260810 04:10:34.456529 21305 heartbeater.cc:347] Connected to a master server at 127.31.32.62:36631
I20260810 04:10:34.456629 21305 heartbeater.cc:464] Registering TS with master...
I20260810 04:10:34.456831 21305 heartbeater.cc:511] Master 127.31.32.62:36631 requested a full tablet report, sending...
I20260810 04:10:34.456817 31872 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskws89Io/build/release/bin/kudu
/tmp/dist-test-taskws89Io/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786334794676589-31872-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786334794676589-31872-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786334794676589-31872-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786334794676589-31872-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.32.2:0
--local_ip_for_outbound_sockets=127.31.32.2
--webserver_interface=127.31.32.2
--webserver_port=0
--tserver_master_addrs=127.31.32.62:36631
--builtin_ntp_servers=127.31.32.20:40207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 04:10:34.457409 21086 ts_manager.cc:194] Registered new tserver with Master: a64e52bddcbe48bf84196986142ba744 (127.31.32.1:33441)
I20260810 04:10:34.458302 21086 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.32.1:42319
W20260810 04:10:34.542724 21308 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:10:34.542938 21308 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:10:34.542960 21308 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:10:34.544040 21308 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:10:34.544076 21308 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:10:34.544101 21308 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.32.2
I20260810 04:10:34.545475 21308 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.32.20:40207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786334794676589-31872-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786334794676589-31872-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.32.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786334794676589-31872-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.32.2
--webserver_port=0
--tserver_master_addrs=127.31.32.62:36631
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21308
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.32.2
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786334794676589-31872-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:10:34.545665 21308 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:10:34.545907 21308 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:10:34.551157 21316 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:10:34.551221 21308 server_base.cc:1034] running on GCE node
W20260810 04:10:34.551223 21314 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:10:34.551172 21313 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:10:34.551708 21308 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:10:34.551985 21308 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:10:34.553133 21308 hybrid_clock.cc:648] HybridClock initialized: now 1786335034553143 us; error 51 us; skew 500 ppm
I20260810 04:10:34.554378 21308 webserver.cc:460] Webserver started at http://127.31.32.2:33293/ using document root <none> and password file <none>
I20260810 04:10:34.554682 21308 fs_manager.cc:359] Metadata directory not provided
I20260810 04:10:34.554733 21308 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:10:34.554925 21308 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:10:34.556210 21308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786334794676589-31872-0/minicluster-data/ts-1/data/instance:
uuid: "cc1e2b65bace4be386a42d7432818d4c"
format_stamp: "Formatted at 2026-08-10 04:10:34 on dist-test-slave-8w9v"
server_key: "0120c8086603bd9970afeac143c30db8"
server_key_iv: "4ccd907a6e0e23d0635743c737f34642"
server_key_version: "encryptionkey@0"
I20260810 04:10:34.556602 21308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786334794676589-31872-0/minicluster-data/ts-1/wal/instance:
uuid: "cc1e2b65bace4be386a42d7432818d4c"
format_stamp: "Formatted at 2026-08-10 04:10:34 on dist-test-slave-8w9v"
server_key: "0120c8086603bd9970afeac143c30db8"
server_key_iv: "4ccd907a6e0e23d0635743c737f34642"
server_key_version: "encryptionkey@0"
I20260810 04:10:34.557952 21308 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260810 04:10:34.558678 21323 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:10:34.558862 21308 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 04:10:34.558974 21308 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786334794676589-31872-0/minicluster-data/ts-1/data,/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786334794676589-31872-0/minicluster-data/ts-1/wal
uuid: "cc1e2b65bace4be386a42d7432818d4c"
format_stamp: "Formatted at 2026-08-10 04:10:34 on dist-test-slave-8w9v"
server_key: "0120c8086603bd9970afeac143c30db8"
server_key_iv: "4ccd907a6e0e23d0635743c737f34642"
server_key_version: "encryptionkey@0"
I20260810 04:10:34.559062 21308 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786334794676589-31872-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786334794676589-31872-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786334794676589-31872-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:10:34.587013 21308 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:10:34.587391 21308 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:10:34.587565 21308 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:10:34.587834 21308 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:10:34.588266 21308 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:10:34.588331 21308 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:10:34.588371 21308 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:10:34.588425 21308 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:10:34.594671 21308 rpc_server.cc:307] RPC server started. Bound to: 127.31.32.2:35005
I20260810 04:10:34.594713 21445 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.32.2:35005 every 8 connection(s)
I20260810 04:10:34.595070 21308 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786334794676589-31872-0/minicluster-data/ts-1/data/info.pb
I20260810 04:10:34.600597 21146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:34.595889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.2:41981 (local address 127.31.32.62:36631)
0810 04:10:34.595963 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0810 04:10:34.595965 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:10:34.596262 (+ 297us) server_negotiation.cc:407] Connection header received
0810 04:10:34.596827 (+ 565us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:10:34.596828 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:10:34.596842 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:10:34.596894 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:10:34.597581 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:34.598098 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:34.599106 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:34.599317 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:34.599413 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:10:34.600291 (+ 878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:10:34.600293 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:10:34.600293 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:10:34.600299 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:10:34.600315 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:10:34.600320 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:10:34.600379 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0810 04:10:34.600482 (+ 103us) server_negotiation.cc:299] Negotiation successful
0810 04:10:34.600514 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260810 04:10:34.600821 21448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:34.595863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:36631 (local address 127.31.32.2:41981)
0810 04:10:34.596105 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0810 04:10:34.596161 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0810 04:10:34.596698 (+ 537us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:10:34.596970 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:10:34.596974 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:10:34.596982 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:10:34.597479 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:10:34.597483 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:10:34.598213 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:10:34.598214 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:10:34.599003 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:10:34.599005 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:10:34.599423 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:10:34.599425 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:10:34.599559 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:10:34.600084 (+ 525us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 04:10:34.600085 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:10:34.600115 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 04:10:34.600196 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:10:34.600381 (+ 185us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:10:34.600387 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:10:34.600390 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 04:10:34.600481 (+ 91us) client_negotiation.cc:240] Negotiation successful
0810 04:10:34.600585 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":94,"threads_started":1}
I20260810 04:10:34.601395 21446 heartbeater.cc:347] Connected to a master server at 127.31.32.62:36631
I20260810 04:10:34.601492 21446 heartbeater.cc:464] Registering TS with master...
I20260810 04:10:34.601637 21446 heartbeater.cc:511] Master 127.31.32.62:36631 requested a full tablet report, sending...
I20260810 04:10:34.602082 21086 ts_manager.cc:194] Registered new tserver with Master: cc1e2b65bace4be386a42d7432818d4c (127.31.32.2:35005)
I20260810 04:10:34.602520 21086 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.32.2:41981
I20260810 04:10:34.603279 31872 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskws89Io/build/release/bin/kudu as pid 21308
I20260810 04:10:34.603407 31872 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786334794676589-31872-0/minicluster-data/ts-1/wal/instance
I20260810 04:10:34.603621 31872 external_mini_cluster.cc:1428] Setting key 2b0ae2224c2997b35a85c0eb69e92792
I20260810 04:10:34.604722 31872 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskws89Io/build/release/bin/kudu
/tmp/dist-test-taskws89Io/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786334794676589-31872-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786334794676589-31872-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786334794676589-31872-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786334794676589-31872-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.32.3:0
--local_ip_for_outbound_sockets=127.31.32.3
--webserver_interface=127.31.32.3
--webserver_port=0
--tserver_master_addrs=127.31.32.62:36631
--builtin_ntp_servers=127.31.32.20:40207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 04:10:34.690609 21449 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:10:34.690841 21449 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:10:34.690865 21449 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:10:34.692055 21449 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:10:34.692104 21449 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:10:34.692199 21449 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.32.3
I20260810 04:10:34.693606 21449 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.32.20:40207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786334794676589-31872-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786334794676589-31872-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.32.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786334794676589-31872-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.32.3
--webserver_port=0
--tserver_master_addrs=127.31.32.62:36631
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21449
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.32.3
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786334794676589-31872-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:10:34.693892 21449 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:10:34.694217 21449 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 04:10:34.699599 21449 server_base.cc:1034] running on GCE node
W20260810 04:10:34.699544 21455 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:10:34.699553 21454 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:10:34.699546 21457 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:10:34.700027 21449 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:10:34.700295 21449 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:10:34.701450 21449 hybrid_clock.cc:648] HybridClock initialized: now 1786335034701436 us; error 57 us; skew 500 ppm
I20260810 04:10:34.702656 21449 webserver.cc:460] Webserver started at http://127.31.32.3:38905/ using document root <none> and password file <none>
I20260810 04:10:34.702929 21449 fs_manager.cc:359] Metadata directory not provided
I20260810 04:10:34.703004 21449 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:10:34.703147 21449 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:10:34.704308 21449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786334794676589-31872-0/minicluster-data/ts-2/data/instance:
uuid: "ec07a344c2ee4f3d84bbcdaabd8971d3"
format_stamp: "Formatted at 2026-08-10 04:10:34 on dist-test-slave-8w9v"
server_key: "b7e003a5bd18a3b9cc3fe54c04d0be52"
server_key_iv: "efe1894ebcf75ce9c68da8030f6b9af6"
server_key_version: "encryptionkey@0"
I20260810 04:10:34.704685 21449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786334794676589-31872-0/minicluster-data/ts-2/wal/instance:
uuid: "ec07a344c2ee4f3d84bbcdaabd8971d3"
format_stamp: "Formatted at 2026-08-10 04:10:34 on dist-test-slave-8w9v"
server_key: "b7e003a5bd18a3b9cc3fe54c04d0be52"
server_key_iv: "efe1894ebcf75ce9c68da8030f6b9af6"
server_key_version: "encryptionkey@0"
I20260810 04:10:34.706074 21449 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260810 04:10:34.706853 21463 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:10:34.707036 21449 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:10:34.707139 21449 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786334794676589-31872-0/minicluster-data/ts-2/data,/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786334794676589-31872-0/minicluster-data/ts-2/wal
uuid: "ec07a344c2ee4f3d84bbcdaabd8971d3"
format_stamp: "Formatted at 2026-08-10 04:10:34 on dist-test-slave-8w9v"
server_key: "b7e003a5bd18a3b9cc3fe54c04d0be52"
server_key_iv: "efe1894ebcf75ce9c68da8030f6b9af6"
server_key_version: "encryptionkey@0"
I20260810 04:10:34.707224 21449 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786334794676589-31872-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786334794676589-31872-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786334794676589-31872-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:10:34.727959 21449 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:10:34.728363 21449 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:10:34.728526 21449 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:10:34.728771 21449 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:10:34.729130 21449 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:10:34.729189 21449 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:10:34.729233 21449 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:10:34.729254 21449 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:10:34.735167 21449 rpc_server.cc:307] RPC server started. Bound to: 127.31.32.3:45563
I20260810 04:10:34.735221 21593 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.32.3:45563 every 8 connection(s)
I20260810 04:10:34.735541 21449 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786334794676589-31872-0/minicluster-data/ts-2/data/info.pb
I20260810 04:10:34.740731 31872 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskws89Io/build/release/bin/kudu as pid 21449
I20260810 04:10:34.740888 31872 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786334794676589-31872-0/minicluster-data/ts-2/wal/instance
I20260810 04:10:34.741098 21146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:34.736424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.3:54985 (local address 127.31.32.62:36631)
0810 04:10:34.736491 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0810 04:10:34.736493 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:10:34.736700 (+ 207us) server_negotiation.cc:407] Connection header received
0810 04:10:34.737190 (+ 490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:10:34.737191 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:10:34.737201 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:10:34.737269 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:10:34.737924 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:34.738419 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:34.739386 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:34.739585 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:34.739645 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:10:34.740702 (+ 1057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:10:34.740703 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:10:34.740704 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:10:34.740709 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:10:34.740726 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:10:34.740730 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:10:34.740797 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0810 04:10:34.740966 (+ 169us) server_negotiation.cc:299] Negotiation successful
0810 04:10:34.741003 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260810 04:10:34.741143 31872 external_mini_cluster.cc:1428] Setting key 9dca298f97328993e615cf662efa9478
I20260810 04:10:34.741521 21597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:34.736381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:36631 (local address 127.31.32.3:54985)
0810 04:10:34.736561 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0810 04:10:34.736604 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 04:10:34.737102 (+ 498us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:10:34.737326 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:10:34.737330 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:10:34.737340 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:10:34.737848 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:10:34.737852 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:10:34.738509 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:10:34.738510 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:10:34.739279 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:10:34.739281 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:10:34.739708 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:10:34.739709 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:10:34.739829 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:10:34.740518 (+ 689us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 04:10:34.740519 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:10:34.740560 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 04:10:34.740591 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:10:34.740824 (+ 233us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:10:34.740846 (+ 22us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:10:34.740850 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 04:10:34.741208 (+ 358us) client_negotiation.cc:240] Negotiation successful
0810 04:10:34.741329 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":86,"threads_started":1}
I20260810 04:10:34.742051 21595 heartbeater.cc:347] Connected to a master server at 127.31.32.62:36631
I20260810 04:10:34.742161 21595 heartbeater.cc:464] Registering TS with master...
I20260810 04:10:34.742357 21595 heartbeater.cc:511] Master 127.31.32.62:36631 requested a full tablet report, sending...
I20260810 04:10:34.742769 21086 ts_manager.cc:194] Registered new tserver with Master: ec07a344c2ee4f3d84bbcdaabd8971d3 (127.31.32.3:45563)
I20260810 04:10:34.743100 21086 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.32.3:54985
I20260810 04:10:34.753269 31872 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 04:10:34.758050 21146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:34.754769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51604 (local address 127.31.32.62:36631)
0810 04:10:34.754851 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0810 04:10:34.754853 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:10:34.754972 (+ 119us) server_negotiation.cc:407] Connection header received
0810 04:10:34.755067 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:10:34.755068 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:10:34.755085 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:10:34.755178 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:10:34.755367 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:34.755860 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:34.756732 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:34.756929 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:34.757046 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:10:34.757800 (+ 754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:10:34.757802 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:10:34.757802 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:10:34.757807 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:10:34.757823 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:10:34.757828 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:10:34.757884 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0810 04:10:34.757937 (+ 53us) server_negotiation.cc:299] Negotiation successful
0810 04:10:34.757963 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 04:10:34.759733 21086 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51604:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 04:10:34.761445 21086 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 04:10:34.768051 21605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:34.763753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.1:33441 (local address 127.0.0.1:57940)
0810 04:10:34.763891 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0810 04:10:34.763933 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 04:10:34.763998 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:10:34.764750 (+ 752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:10:34.764751 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:10:34.764752 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:10:34.764785 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:10:34.764786 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:10:34.766024 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:10:34.766026 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:10:34.767015 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:10:34.767019 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:10:34.767755 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:10:34.767756 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:10:34.767842 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:10:34.767856 (+ 14us) client_negotiation.cc:769] Sending connection context
0810 04:10:34.767922 (+ 66us) client_negotiation.cc:240] Negotiation successful
0810 04:10:34.767974 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":69,"threads_started":1}
I20260810 04:10:34.768349 21606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:34.763887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57940 (local address 127.31.32.1:33441)
0810 04:10:34.764598 (+ 711us) server_negotiation.cc:206] Beginning negotiation
0810 04:10:34.764603 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 04:10:34.764613 (+ 10us) server_negotiation.cc:407] Connection header received
0810 04:10:34.764663 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:10:34.764665 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:10:34.764679 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:10:34.764755 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:10:34.764862 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:34.765928 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:34.767117 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:34.767678 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:34.767753 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:10:34.767889 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0810 04:10:34.767937 (+ 48us) server_negotiation.cc:299] Negotiation successful
0810 04:10:34.767974 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":670,"thread_start_us":85,"threads_started":1}
I20260810 04:10:34.768435 21604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:34.763777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.2:35005 (local address 127.0.0.1:57662)
0810 04:10:34.764059 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0810 04:10:34.764072 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 04:10:34.764177 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:10:34.764407 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:10:34.764411 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:10:34.764414 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:10:34.764485 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:10:34.764487 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:10:34.765790 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:10:34.765791 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:10:34.767099 (+ 1308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:10:34.767100 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:10:34.768151 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:10:34.768152 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:10:34.768220 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:10:34.768226 (+ 6us) client_negotiation.cc:769] Sending connection context
0810 04:10:34.768278 (+ 52us) client_negotiation.cc:240] Negotiation successful
0810 04:10:34.768356 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"mutex_wait_us":95,"thread_start_us":57,"threads_started":1}
I20260810 04:10:34.768541 21609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:34.764203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36918 (local address 127.31.32.3:45563)
0810 04:10:34.764533 (+ 330us) server_negotiation.cc:206] Beginning negotiation
0810 04:10:34.764537 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 04:10:34.764548 (+ 11us) server_negotiation.cc:407] Connection header received
0810 04:10:34.764585 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:10:34.764587 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:10:34.764603 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:10:34.764864 (+ 261us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:10:34.764922 (+ 58us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:34.765599 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:34.767201 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:34.767939 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:34.768010 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:10:34.768088 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0810 04:10:34.768400 (+ 312us) server_negotiation.cc:299] Negotiation successful
0810 04:10:34.768438 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":86,"threads_started":1}
I20260810 04:10:34.768649 21607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:34.763968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.3:45563 (local address 127.0.0.1:36918)
0810 04:10:34.764419 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0810 04:10:34.764429 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 04:10:34.764476 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:10:34.764731 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:10:34.764732 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:10:34.764734 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:10:34.764784 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:10:34.764786 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:10:34.765694 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:10:34.765695 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:10:34.767091 (+ 1396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:10:34.767094 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:10:34.768149 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:10:34.768151 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:10:34.768220 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:10:34.768226 (+ 6us) client_negotiation.cc:769] Sending connection context
0810 04:10:34.768278 (+ 52us) client_negotiation.cc:240] Negotiation successful
0810 04:10:34.768354 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":435,"mutex_wait_us":38,"thread_start_us":68,"threads_started":1}
I20260810 04:10:34.769053 21608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:34.763987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57662 (local address 127.31.32.2:35005)
0810 04:10:34.764187 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0810 04:10:34.764191 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 04:10:34.764203 (+ 12us) server_negotiation.cc:407] Connection header received
0810 04:10:34.764253 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:10:34.764254 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:10:34.764266 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:10:34.764375 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:10:34.764578 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:34.765721 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:34.767201 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:34.767939 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:34.768009 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:10:34.768088 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0810 04:10:34.768933 (+ 845us) server_negotiation.cc:299] Negotiation successful
0810 04:10:34.768977 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":112,"threads_started":1}
I20260810 04:10:34.769155 21230 tablet_service.cc:1467] Processing CreateTablet for tablet 75df8ccea26047d7a31c7aaf6a827bd9 (DEFAULT_TABLE table=test-table [id=11e8fd0558fd43899de17be694b29b24]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:10:34.769570 21230 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75df8ccea26047d7a31c7aaf6a827bd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:10:34.769739 21372 tablet_service.cc:1467] Processing CreateTablet for tablet 75df8ccea26047d7a31c7aaf6a827bd9 (DEFAULT_TABLE table=test-table [id=11e8fd0558fd43899de17be694b29b24]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:10:34.770087 21372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75df8ccea26047d7a31c7aaf6a827bd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:10:34.772920 21610 tablet_bootstrap.cc:492] T 75df8ccea26047d7a31c7aaf6a827bd9 P a64e52bddcbe48bf84196986142ba744: Bootstrap starting.
I20260810 04:10:34.773283 21611 tablet_bootstrap.cc:492] T 75df8ccea26047d7a31c7aaf6a827bd9 P cc1e2b65bace4be386a42d7432818d4c: Bootstrap starting.
I20260810 04:10:34.773289 21514 tablet_service.cc:1467] Processing CreateTablet for tablet 75df8ccea26047d7a31c7aaf6a827bd9 (DEFAULT_TABLE table=test-table [id=11e8fd0558fd43899de17be694b29b24]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:10:34.773672 21514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75df8ccea26047d7a31c7aaf6a827bd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:10:34.773751 21610 tablet_bootstrap.cc:654] T 75df8ccea26047d7a31c7aaf6a827bd9 P a64e52bddcbe48bf84196986142ba744: Neither blocks nor log segments found. Creating new log.
I20260810 04:10:34.774019 21611 tablet_bootstrap.cc:654] T 75df8ccea26047d7a31c7aaf6a827bd9 P cc1e2b65bace4be386a42d7432818d4c: Neither blocks nor log segments found. Creating new log.
I20260810 04:10:34.774125 21610 log.cc:824] T 75df8ccea26047d7a31c7aaf6a827bd9 P a64e52bddcbe48bf84196986142ba744: Log is configured to *not* fsync() on all Append() calls
I20260810 04:10:34.774379 21611 log.cc:824] T 75df8ccea26047d7a31c7aaf6a827bd9 P cc1e2b65bace4be386a42d7432818d4c: Log is configured to *not* fsync() on all Append() calls
I20260810 04:10:34.775019 21610 tablet_bootstrap.cc:492] T 75df8ccea26047d7a31c7aaf6a827bd9 P a64e52bddcbe48bf84196986142ba744: No bootstrap required, opened a new log
I20260810 04:10:34.775120 21610 ts_tablet_manager.cc:1397] T 75df8ccea26047d7a31c7aaf6a827bd9 P a64e52bddcbe48bf84196986142ba744: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 04:10:34.775230 21611 tablet_bootstrap.cc:492] T 75df8ccea26047d7a31c7aaf6a827bd9 P cc1e2b65bace4be386a42d7432818d4c: No bootstrap required, opened a new log
I20260810 04:10:34.775303 21611 ts_tablet_manager.cc:1397] T 75df8ccea26047d7a31c7aaf6a827bd9 P cc1e2b65bace4be386a42d7432818d4c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 04:10:34.777206 21614 tablet_bootstrap.cc:492] T 75df8ccea26047d7a31c7aaf6a827bd9 P ec07a344c2ee4f3d84bbcdaabd8971d3: Bootstrap starting.
I20260810 04:10:34.777590 21610 raft_consensus.cc:348] T 75df8ccea26047d7a31c7aaf6a827bd9 P a64e52bddcbe48bf84196986142ba744 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64e52bddcbe48bf84196986142ba744" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 33441 } } peers { permanent_uuid: "cc1e2b65bace4be386a42d7432818d4c" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 35005 } } peers { permanent_uuid: "ec07a344c2ee4f3d84bbcdaabd8971d3" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 45563 } }
I20260810 04:10:34.777794 21610 raft_consensus.cc:374] T 75df8ccea26047d7a31c7aaf6a827bd9 P a64e52bddcbe48bf84196986142ba744 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:10:34.777848 21610 raft_consensus.cc:729] T 75df8ccea26047d7a31c7aaf6a827bd9 P a64e52bddcbe48bf84196986142ba744 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64e52bddcbe48bf84196986142ba744, State: Initialized, Role: FOLLOWER
I20260810 04:10:34.777978 21614 tablet_bootstrap.cc:654] T 75df8ccea26047d7a31c7aaf6a827bd9 P ec07a344c2ee4f3d84bbcdaabd8971d3: Neither blocks nor log segments found. Creating new log.
I20260810 04:10:34.778009 21610 consensus_queue.cc:260] T 75df8ccea26047d7a31c7aaf6a827bd9 P a64e52bddcbe48bf84196986142ba744 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64e52bddcbe48bf84196986142ba744" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 33441 } } peers { permanent_uuid: "cc1e2b65bace4be386a42d7432818d4c" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 35005 } } peers { permanent_uuid: "ec07a344c2ee4f3d84bbcdaabd8971d3" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 45563 } }
I20260810 04:10:34.778052 21611 raft_consensus.cc:348] T 75df8ccea26047d7a31c7aaf6a827bd9 P cc1e2b65bace4be386a42d7432818d4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64e52bddcbe48bf84196986142ba744" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 33441 } } peers { permanent_uuid: "cc1e2b65bace4be386a42d7432818d4c" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 35005 } } peers { permanent_uuid: "ec07a344c2ee4f3d84bbcdaabd8971d3" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 45563 } }
I20260810 04:10:34.778218 21611 raft_consensus.cc:374] T 75df8ccea26047d7a31c7aaf6a827bd9 P cc1e2b65bace4be386a42d7432818d4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:10:34.778263 21611 raft_consensus.cc:729] T 75df8ccea26047d7a31c7aaf6a827bd9 P cc1e2b65bace4be386a42d7432818d4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc1e2b65bace4be386a42d7432818d4c, State: Initialized, Role: FOLLOWER
I20260810 04:10:34.778343 21610 ts_tablet_manager.cc:1428] T 75df8ccea26047d7a31c7aaf6a827bd9 P a64e52bddcbe48bf84196986142ba744: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260810 04:10:34.778344 21614 log.cc:824] T 75df8ccea26047d7a31c7aaf6a827bd9 P ec07a344c2ee4f3d84bbcdaabd8971d3: Log is configured to *not* fsync() on all Append() calls
I20260810 04:10:34.778394 21611 consensus_queue.cc:260] T 75df8ccea26047d7a31c7aaf6a827bd9 P cc1e2b65bace4be386a42d7432818d4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64e52bddcbe48bf84196986142ba744" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 33441 } } peers { permanent_uuid: "cc1e2b65bace4be386a42d7432818d4c" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 35005 } } peers { permanent_uuid: "ec07a344c2ee4f3d84bbcdaabd8971d3" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 45563 } }
I20260810 04:10:34.778532 21305 heartbeater.cc:503] Master 127.31.32.62:36631 was elected leader, sending a full tablet report...
I20260810 04:10:34.778662 21611 ts_tablet_manager.cc:1428] T 75df8ccea26047d7a31c7aaf6a827bd9 P cc1e2b65bace4be386a42d7432818d4c: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260810 04:10:34.778689 21446 heartbeater.cc:503] Master 127.31.32.62:36631 was elected leader, sending a full tablet report...
I20260810 04:10:34.779868 21614 tablet_bootstrap.cc:492] T 75df8ccea26047d7a31c7aaf6a827bd9 P ec07a344c2ee4f3d84bbcdaabd8971d3: No bootstrap required, opened a new log
I20260810 04:10:34.779947 21614 ts_tablet_manager.cc:1397] T 75df8ccea26047d7a31c7aaf6a827bd9 P ec07a344c2ee4f3d84bbcdaabd8971d3: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 04:10:34.782003 21614 raft_consensus.cc:348] T 75df8ccea26047d7a31c7aaf6a827bd9 P ec07a344c2ee4f3d84bbcdaabd8971d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64e52bddcbe48bf84196986142ba744" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 33441 } } peers { permanent_uuid: "cc1e2b65bace4be386a42d7432818d4c" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 35005 } } peers { permanent_uuid: "ec07a344c2ee4f3d84bbcdaabd8971d3" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 45563 } }
I20260810 04:10:34.782168 21614 raft_consensus.cc:374] T 75df8ccea26047d7a31c7aaf6a827bd9 P ec07a344c2ee4f3d84bbcdaabd8971d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:10:34.782218 21614 raft_consensus.cc:729] T 75df8ccea26047d7a31c7aaf6a827bd9 P ec07a344c2ee4f3d84bbcdaabd8971d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec07a344c2ee4f3d84bbcdaabd8971d3, State: Initialized, Role: FOLLOWER
I20260810 04:10:34.782343 21614 consensus_queue.cc:260] T 75df8ccea26047d7a31c7aaf6a827bd9 P ec07a344c2ee4f3d84bbcdaabd8971d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64e52bddcbe48bf84196986142ba744" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 33441 } } peers { permanent_uuid: "cc1e2b65bace4be386a42d7432818d4c" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 35005 } } peers { permanent_uuid: "ec07a344c2ee4f3d84bbcdaabd8971d3" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 45563 } }
I20260810 04:10:34.782621 21614 ts_tablet_manager.cc:1428] T 75df8ccea26047d7a31c7aaf6a827bd9 P ec07a344c2ee4f3d84bbcdaabd8971d3: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 04:10:34.782745 21595 heartbeater.cc:503] Master 127.31.32.62:36631 was elected leader, sending a full tablet report...
W20260810 04:10:34.845696 21447 tablet.cc:2369] T 75df8ccea26047d7a31c7aaf6a827bd9 P cc1e2b65bace4be386a42d7432818d4c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 04:10:34.896435 21616 raft_consensus.cc:482] T 75df8ccea26047d7a31c7aaf6a827bd9 P cc1e2b65bace4be386a42d7432818d4c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 04:10:34.896559 21616 raft_consensus.cc:504] T 75df8ccea26047d7a31c7aaf6a827bd9 P cc1e2b65bace4be386a42d7432818d4c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64e52bddcbe48bf84196986142ba744" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 33441 } } peers { permanent_uuid: "cc1e2b65bace4be386a42d7432818d4c" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 35005 } } peers { permanent_uuid: "ec07a344c2ee4f3d84bbcdaabd8971d3" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 45563 } }
I20260810 04:10:34.896963 21616 leader_election.cc:288] T 75df8ccea26047d7a31c7aaf6a827bd9 P cc1e2b65bace4be386a42d7432818d4c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a64e52bddcbe48bf84196986142ba744 (127.31.32.1:33441), ec07a344c2ee4f3d84bbcdaabd8971d3 (127.31.32.3:45563)
I20260810 04:10:34.899998 21609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:34.897492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.2:60575 (local address 127.31.32.3:45563)
0810 04:10:34.897557 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0810 04:10:34.897559 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:10:34.897570 (+ 11us) server_negotiation.cc:407] Connection header received
0810 04:10:34.897621 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:10:34.897622 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:10:34.897632 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:10:34.897709 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:10:34.897888 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:34.898361 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:34.899287 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:34.899631 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:34.899721 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:10:34.899731 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0810 04:10:34.899890 (+ 159us) server_negotiation.cc:299] Negotiation successful
0810 04:10:34.899925 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260810 04:10:34.900022 21606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:34.897279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.32.2:41257 (local address 127.31.32.1:33441)
0810 04:10:34.897357 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0810 04:10:34.897359 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:10:34.897371 (+ 12us) server_negotiation.cc:407] Connection header received
0810 04:10:34.897408 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:10:34.897409 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:10:34.897419 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:10:34.897518 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:10:34.897725 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:34.898220 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:34.899264 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:34.899633 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:34.899738 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:10:34.899747 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0810 04:10:34.899911 (+ 164us) server_negotiation.cc:299] Negotiation successful
0810 04:10:34.899952 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 04:10:34.900022 21448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:34.897179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.1:33441 (local address 127.31.32.2:41257)
0810 04:10:34.897245 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0810 04:10:34.897259 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 04:10:34.897310 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:10:34.897555 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:10:34.897557 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:10:34.897559 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:10:34.897635 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:10:34.897637 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:10:34.898311 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:10:34.898312 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:10:34.899135 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:10:34.899138 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:10:34.899742 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:10:34.899743 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:10:34.899820 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:10:34.899823 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 04:10:34.899892 (+ 69us) client_negotiation.cc:240] Negotiation successful
0810 04:10:34.899935 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260810 04:10:34.900024 21621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:34.897328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.3:45563 (local address 127.31.32.2:60575)
0810 04:10:34.897504 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0810 04:10:34.897514 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 04:10:34.897564 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:10:34.897713 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:10:34.897714 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:10:34.897715 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:10:34.897808 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:10:34.897810 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:10:34.898452 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:10:34.898453 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:10:34.899182 (+ 729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:10:34.899184 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:10:34.899722 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:10:34.899724 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:10:34.899807 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:10:34.899811 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 04:10:34.899883 (+ 72us) client_negotiation.cc:240] Negotiation successful
0810 04:10:34.899924 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":107,"threads_started":1}
I20260810 04:10:34.900413 21253 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75df8ccea26047d7a31c7aaf6a827bd9" candidate_uuid: "cc1e2b65bace4be386a42d7432818d4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64e52bddcbe48bf84196986142ba744" is_pre_election: true
I20260810 04:10:34.900607 21253 raft_consensus.cc:2454] T 75df8ccea26047d7a31c7aaf6a827bd9 P a64e52bddcbe48bf84196986142ba744 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc1e2b65bace4be386a42d7432818d4c in term 0.
I20260810 04:10:34.900897 21327 leader_election.cc:302] T 75df8ccea26047d7a31c7aaf6a827bd9 P cc1e2b65bace4be386a42d7432818d4c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64e52bddcbe48bf84196986142ba744, cc1e2b65bace4be386a42d7432818d4c; no voters:
I20260810 04:10:34.900882 21537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75df8ccea26047d7a31c7aaf6a827bd9" candidate_uuid: "cc1e2b65bace4be386a42d7432818d4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec07a344c2ee4f3d84bbcdaabd8971d3" is_pre_election: true
I20260810 04:10:34.901054 21537 raft_consensus.cc:2454] T 75df8ccea26047d7a31c7aaf6a827bd9 P ec07a344c2ee4f3d84bbcdaabd8971d3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc1e2b65bace4be386a42d7432818d4c in term 0.
I20260810 04:10:34.901072 21616 raft_consensus.cc:2781] T 75df8ccea26047d7a31c7aaf6a827bd9 P cc1e2b65bace4be386a42d7432818d4c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 04:10:34.901139 21616 raft_consensus.cc:482] T 75df8ccea26047d7a31c7aaf6a827bd9 P cc1e2b65bace4be386a42d7432818d4c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 04:10:34.901292 21616 raft_consensus.cc:3037] T 75df8ccea26047d7a31c7aaf6a827bd9 P cc1e2b65bace4be386a42d7432818d4c [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:10:34.902115 21616 raft_consensus.cc:504] T 75df8ccea26047d7a31c7aaf6a827bd9 P cc1e2b65bace4be386a42d7432818d4c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64e52bddcbe48bf84196986142ba744" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 33441 } } peers { permanent_uuid: "cc1e2b65bace4be386a42d7432818d4c" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 35005 } } peers { permanent_uuid: "ec07a344c2ee4f3d84bbcdaabd8971d3" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 45563 } }
I20260810 04:10:34.902258 21616 leader_election.cc:288] T 75df8ccea26047d7a31c7aaf6a827bd9 P cc1e2b65bace4be386a42d7432818d4c [CANDIDATE]: Term 1 election: Requested vote from peers a64e52bddcbe48bf84196986142ba744 (127.31.32.1:33441), ec07a344c2ee4f3d84bbcdaabd8971d3 (127.31.32.3:45563)
I20260810 04:10:34.902516 21253 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75df8ccea26047d7a31c7aaf6a827bd9" candidate_uuid: "cc1e2b65bace4be386a42d7432818d4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64e52bddcbe48bf84196986142ba744"
I20260810 04:10:34.902522 21537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75df8ccea26047d7a31c7aaf6a827bd9" candidate_uuid: "cc1e2b65bace4be386a42d7432818d4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec07a344c2ee4f3d84bbcdaabd8971d3"
I20260810 04:10:34.902616 21253 raft_consensus.cc:3037] T 75df8ccea26047d7a31c7aaf6a827bd9 P a64e52bddcbe48bf84196986142ba744 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:10:34.902612 21537 raft_consensus.cc:3037] T 75df8ccea26047d7a31c7aaf6a827bd9 P ec07a344c2ee4f3d84bbcdaabd8971d3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:10:34.903457 21253 raft_consensus.cc:2454] T 75df8ccea26047d7a31c7aaf6a827bd9 P a64e52bddcbe48bf84196986142ba744 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc1e2b65bace4be386a42d7432818d4c in term 1.
I20260810 04:10:34.903465 21537 raft_consensus.cc:2454] T 75df8ccea26047d7a31c7aaf6a827bd9 P ec07a344c2ee4f3d84bbcdaabd8971d3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc1e2b65bace4be386a42d7432818d4c in term 1.
I20260810 04:10:34.903695 21327 leader_election.cc:302] T 75df8ccea26047d7a31c7aaf6a827bd9 P cc1e2b65bace4be386a42d7432818d4c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64e52bddcbe48bf84196986142ba744, cc1e2b65bace4be386a42d7432818d4c; no voters:
I20260810 04:10:34.903832 21616 raft_consensus.cc:2781] T 75df8ccea26047d7a31c7aaf6a827bd9 P cc1e2b65bace4be386a42d7432818d4c [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:10:34.904027 21616 raft_consensus.cc:686] T 75df8ccea26047d7a31c7aaf6a827bd9 P cc1e2b65bace4be386a42d7432818d4c [term 1 LEADER]: Becoming Leader. State: Replica: cc1e2b65bace4be386a42d7432818d4c, State: Running, Role: LEADER
I20260810 04:10:34.904114 21616 consensus_queue.cc:237] T 75df8ccea26047d7a31c7aaf6a827bd9 P cc1e2b65bace4be386a42d7432818d4c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64e52bddcbe48bf84196986142ba744" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 33441 } } peers { permanent_uuid: "cc1e2b65bace4be386a42d7432818d4c" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 35005 } } peers { permanent_uuid: "ec07a344c2ee4f3d84bbcdaabd8971d3" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 45563 } }
I20260810 04:10:34.905072 21085 catalog_manager.cc:5495] T 75df8ccea26047d7a31c7aaf6a827bd9 P cc1e2b65bace4be386a42d7432818d4c reported cstate change: term changed from 0 to 1, leader changed from <none> to cc1e2b65bace4be386a42d7432818d4c (127.31.32.2). New cstate: current_term: 1 leader_uuid: "cc1e2b65bace4be386a42d7432818d4c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64e52bddcbe48bf84196986142ba744" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 33441 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cc1e2b65bace4be386a42d7432818d4c" member_type: VOTER last_known_addr { host: "127.31.32.2" port: 35005 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec07a344c2ee4f3d84bbcdaabd8971d3" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 45563 } health_report { overall_health: UNKNOWN } } }
I20260810 04:10:34.913203 21608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:34.910502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57670 (local address 127.31.32.2:35005)
0810 04:10:34.910570 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0810 04:10:34.910572 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:10:34.910608 (+ 36us) server_negotiation.cc:407] Connection header received
0810 04:10:34.910698 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:10:34.910699 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:10:34.910718 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:10:34.910812 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 04:10:34.910960 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:34.911526 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:34.912392 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:34.912608 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:34.912690 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:10:34.912837 (+ 147us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 04:10:34.912944 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 04:10:34.913061 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0810 04:10:34.913081 (+ 20us) server_negotiation.cc:299] Negotiation successful
0810 04:10:34.913103 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260810 04:10:34.915077 21537 raft_consensus.cc:1261] T 75df8ccea26047d7a31c7aaf6a827bd9 P ec07a344c2ee4f3d84bbcdaabd8971d3 [term 1 FOLLOWER]: Refusing update from remote peer cc1e2b65bace4be386a42d7432818d4c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:10:34.915156 21253 raft_consensus.cc:1261] T 75df8ccea26047d7a31c7aaf6a827bd9 P a64e52bddcbe48bf84196986142ba744 [term 1 FOLLOWER]: Refusing update from remote peer cc1e2b65bace4be386a42d7432818d4c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:10:34.915500 21616 consensus_queue.cc:1035] T 75df8ccea26047d7a31c7aaf6a827bd9 P cc1e2b65bace4be386a42d7432818d4c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec07a344c2ee4f3d84bbcdaabd8971d3" member_type: VOTER last_known_addr { host: "127.31.32.3" port: 45563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:10:34.915648 21622 consensus_queue.cc:1035] T 75df8ccea26047d7a31c7aaf6a827bd9 P cc1e2b65bace4be386a42d7432818d4c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64e52bddcbe48bf84196986142ba744" member_type: VOTER last_known_addr { host: "127.31.32.1" port: 33441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:10:34.918196 21627 mvcc.cc:205] Tried to move back new op lower bound from 7316828303009181696 to 7316828302968184832. Current Snapshot: MvccSnapshot[applied={T|T < 7316828303009181696}]
I20260810 04:10:34.918249 21626 mvcc.cc:205] Tried to move back new op lower bound from 7316828303009181696 to 7316828302968184832. Current Snapshot: MvccSnapshot[applied={T|T < 7316828303009181696}]
I20260810 04:10:34.920357 21085 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51604:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 04:10:34.920439 21085 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51604:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 04:10:34.921231 21085 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4a49c1b253d746aeb3699de0c1505ce6: Sending DeleteTablet for 3 replicas of tablet 75df8ccea26047d7a31c7aaf6a827bd9
I20260810 04:10:34.921641 21514 tablet_service.cc:1514] Processing DeleteTablet for tablet 75df8ccea26047d7a31c7aaf6a827bd9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:10:34 UTC) from {username='slave'} at 127.0.0.1:36918
I20260810 04:10:34.921640 21372 tablet_service.cc:1514] Processing DeleteTablet for tablet 75df8ccea26047d7a31c7aaf6a827bd9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:10:34 UTC) from {username='slave'} at 127.0.0.1:57662
I20260810 04:10:34.921869 21230 tablet_service.cc:1514] Processing DeleteTablet for tablet 75df8ccea26047d7a31c7aaf6a827bd9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:10:34 UTC) from {username='slave'} at 127.0.0.1:57940
I20260810 04:10:34.921892 21635 tablet_replica.cc:330] T 75df8ccea26047d7a31c7aaf6a827bd9 P cc1e2b65bace4be386a42d7432818d4c: stopping tablet replica
I20260810 04:10:34.922016 21635 raft_consensus.cc:2229] T 75df8ccea26047d7a31c7aaf6a827bd9 P cc1e2b65bace4be386a42d7432818d4c [term 1 LEADER]: Raft consensus shutting down.
I20260810 04:10:34.922014 31872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskws89Io/build/release/bin/kudu with pid 21161
I20260810 04:10:34.922125 21635 raft_consensus.cc:2258] T 75df8ccea26047d7a31c7aaf6a827bd9 P cc1e2b65bace4be386a42d7432818d4c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:10:34.922602 21636 tablet_replica.cc:330] T 75df8ccea26047d7a31c7aaf6a827bd9 P ec07a344c2ee4f3d84bbcdaabd8971d3: stopping tablet replica
I20260810 04:10:34.922744 21636 raft_consensus.cc:2229] T 75df8ccea26047d7a31c7aaf6a827bd9 P ec07a344c2ee4f3d84bbcdaabd8971d3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 04:10:34.922876 21636 raft_consensus.cc:2258] T 75df8ccea26047d7a31c7aaf6a827bd9 P ec07a344c2ee4f3d84bbcdaabd8971d3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:10:34.923930 21636 ts_tablet_manager.cc:1905] T 75df8ccea26047d7a31c7aaf6a827bd9 P ec07a344c2ee4f3d84bbcdaabd8971d3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:10:34.926152 21636 ts_tablet_manager.cc:1918] T 75df8ccea26047d7a31c7aaf6a827bd9 P ec07a344c2ee4f3d84bbcdaabd8971d3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 04:10:34.926244 21636 log.cc:1196] T 75df8ccea26047d7a31c7aaf6a827bd9 P ec07a344c2ee4f3d84bbcdaabd8971d3: Deleting WAL directory at /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786334794676589-31872-0/minicluster-data/ts-2/wal/wals/75df8ccea26047d7a31c7aaf6a827bd9
I20260810 04:10:34.926566 21636 ts_tablet_manager.cc:1939] T 75df8ccea26047d7a31c7aaf6a827bd9 P ec07a344c2ee4f3d84bbcdaabd8971d3: Deleting consensus metadata
I20260810 04:10:34.927081 21073 catalog_manager.cc:4841] TS ec07a344c2ee4f3d84bbcdaabd8971d3 (127.31.32.3:45563): tablet 75df8ccea26047d7a31c7aaf6a827bd9 (table test-table [id=11e8fd0558fd43899de17be694b29b24]) successfully deleted
I20260810 04:10:34.929811 21635 ts_tablet_manager.cc:1905] T 75df8ccea26047d7a31c7aaf6a827bd9 P cc1e2b65bace4be386a42d7432818d4c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:10:34.931833 21635 ts_tablet_manager.cc:1918] T 75df8ccea26047d7a31c7aaf6a827bd9 P cc1e2b65bace4be386a42d7432818d4c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 04:10:34.931912 21635 log.cc:1196] T 75df8ccea26047d7a31c7aaf6a827bd9 P cc1e2b65bace4be386a42d7432818d4c: Deleting WAL directory at /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786334794676589-31872-0/minicluster-data/ts-1/wal/wals/75df8ccea26047d7a31c7aaf6a827bd9
I20260810 04:10:34.932183 21635 ts_tablet_manager.cc:1939] T 75df8ccea26047d7a31c7aaf6a827bd9 P cc1e2b65bace4be386a42d7432818d4c: Deleting consensus metadata
I20260810 04:10:34.932578 21071 catalog_manager.cc:4841] TS cc1e2b65bace4be386a42d7432818d4c (127.31.32.2:35005): tablet 75df8ccea26047d7a31c7aaf6a827bd9 (table test-table [id=11e8fd0558fd43899de17be694b29b24]) successfully deleted
W20260810 04:10:34.933502 21073 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.32.1:33441 (error 108)
I20260810 04:10:34.933987 21607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:34.933838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.1:33441 (local address 127.0.0.1:57946)
0810 04:10:34.933904 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0810 04:10:34.933947 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.1:33441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260810 04:10:34.934092 21073 catalog_manager.cc:4584] TS a64e52bddcbe48bf84196986142ba744 (127.31.32.1:33441): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 75df8ccea26047d7a31c7aaf6a827bd9: Network error: Client connection negotiation failed: client connection to 127.31.32.1:33441: connect: Connection refused (error 111)
I20260810 04:10:34.934425 31872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskws89Io/build/release/bin/kudu with pid 21308
I20260810 04:10:34.942363 31872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskws89Io/build/release/bin/kudu with pid 21449
I20260810 04:10:34.952611 31872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskws89Io/build/release/bin/kudu with pid 21052
2026-08-10T04:10:34Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (818 ms)
[----------] 6 tests from EncryptionPolicyTest (8948 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786334794676589-31872-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 04:10:35 dist-test-slave-8w9v krb5kdc[21642](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 04:10:35 dist-test-slave-8w9v krb5kdc[21642](info): set up 2 sockets
Aug 10 04:10:35 dist-test-slave-8w9v krb5kdc[21642](info): commencing operation
krb5kdc: starting...
W20260810 04:10:36.053072 31872 mini_kdc.cc:121] Time spent starting KDC: real 1.079s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786334794676589-31872-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786334794676589-31872-0/krb5kdc/test-user.keytab.
Aug 10 04:10:36 dist-test-slave-8w9v krb5kdc[21642](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335036, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T04:10:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T04:10:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786334794676589-31872-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786334794676589-31872-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 04:10:36.175304 31872 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskws89Io/build/release/bin/kudu
/tmp/dist-test-taskws89Io/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786334794676589-31872-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786334794676589-31872-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786334794676589-31872-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786334794676589-31872-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38387
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786334794676589-31872-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38387
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786334794676589-31872-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786334794676589-31872-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786334794676589-31872-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:10:36.258682 21656 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:10:36.258937 21656 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:10:36.258977 21656 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:10:36.260151 21656 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:10:36.260232 21656 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:10:36.260257 21656 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:10:36.260291 21656 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:10:36.260318 21656 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:10:36.261806 21656 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786334794676589-31872-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786334794676589-31872-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38387
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786334794676589-31872-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38387
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786334794676589-31872-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21656
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786334794676589-31872-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:10:36.262094 21656 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:10:36.262336 21656 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:10:36.267450 21661 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:10:36.267485 21656 server_base.cc:1034] running on GCE node
W20260810 04:10:36.267540 21662 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:10:36.267455 21664 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:10:36.267820 21656 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:10:36.268208 21656 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:10:36.269383 21656 hybrid_clock.cc:648] HybridClock initialized: now 1786335036269353 us; error 47 us; skew 500 ppm
Aug 10 04:10:36 dist-test-slave-8w9v krb5kdc[21642](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335036, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:10:36.271984 21656 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 04:10:36.272481 21656 webserver.cc:460] Webserver started at http://127.0.0.1:38655/ using document root <none> and password file <none>
I20260810 04:10:36.272703 21656 fs_manager.cc:359] Metadata directory not provided
I20260810 04:10:36.272744 21656 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:10:36.272830 21656 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:10:36.273839 21656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786334794676589-31872-0/minicluster-data/master-0/data/instance:
uuid: "9536cb4469344db0b7d846da1abf37b7"
format_stamp: "Formatted at 2026-08-10 04:10:36 on dist-test-slave-8w9v"
server_key: "8f9c88de47891ea09037bae5c7d27610"
server_key_iv: "7e0b9eeaf2af13eb4d5235d64adc299a"
server_key_version: "encryptionkey@0"
I20260810 04:10:36.274145 21656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786334794676589-31872-0/minicluster-data/master-0/wal/instance:
uuid: "9536cb4469344db0b7d846da1abf37b7"
format_stamp: "Formatted at 2026-08-10 04:10:36 on dist-test-slave-8w9v"
server_key: "8f9c88de47891ea09037bae5c7d27610"
server_key_iv: "7e0b9eeaf2af13eb4d5235d64adc299a"
server_key_version: "encryptionkey@0"
I20260810 04:10:36.275461 21656 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 04:10:36.276247 21672 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:10:36.276463 21656 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:10:36.276571 21656 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786334794676589-31872-0/minicluster-data/master-0/data,/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786334794676589-31872-0/minicluster-data/master-0/wal
uuid: "9536cb4469344db0b7d846da1abf37b7"
format_stamp: "Formatted at 2026-08-10 04:10:36 on dist-test-slave-8w9v"
server_key: "8f9c88de47891ea09037bae5c7d27610"
server_key_iv: "7e0b9eeaf2af13eb4d5235d64adc299a"
server_key_version: "encryptionkey@0"
I20260810 04:10:36.276656 21656 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786334794676589-31872-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786334794676589-31872-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786334794676589-31872-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:10:36.303340 21656 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:10:36.305449 21656 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:10:36.305629 21656 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:10:36.310364 21656 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38387
I20260810 04:10:36.310416 21727 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38387 every 8 connection(s)
I20260810 04:10:36.310758 21656 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786334794676589-31872-0/minicluster-data/master-0/data/info.pb
I20260810 04:10:36.311697 21728 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:10:36.314288 21728 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9536cb4469344db0b7d846da1abf37b7: Bootstrap starting.
I20260810 04:10:36.315157 21728 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9536cb4469344db0b7d846da1abf37b7: Neither blocks nor log segments found. Creating new log.
I20260810 04:10:36.315493 21728 log.cc:824] T 00000000000000000000000000000000 P 9536cb4469344db0b7d846da1abf37b7: Log is configured to *not* fsync() on all Append() calls
I20260810 04:10:36.316332 21728 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9536cb4469344db0b7d846da1abf37b7: No bootstrap required, opened a new log
I20260810 04:10:36.317927 21728 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9536cb4469344db0b7d846da1abf37b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9536cb4469344db0b7d846da1abf37b7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38387 } }
I20260810 04:10:36.318078 21728 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9536cb4469344db0b7d846da1abf37b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:10:36.318147 21728 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9536cb4469344db0b7d846da1abf37b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9536cb4469344db0b7d846da1abf37b7, State: Initialized, Role: FOLLOWER
I20260810 04:10:36.318307 21728 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9536cb4469344db0b7d846da1abf37b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9536cb4469344db0b7d846da1abf37b7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38387 } }
I20260810 04:10:36.318398 21728 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9536cb4469344db0b7d846da1abf37b7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 04:10:36.318456 21728 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9536cb4469344db0b7d846da1abf37b7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 04:10:36.318507 21728 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9536cb4469344db0b7d846da1abf37b7 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:10:36.319062 21728 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9536cb4469344db0b7d846da1abf37b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9536cb4469344db0b7d846da1abf37b7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38387 } }
I20260810 04:10:36.319187 21728 leader_election.cc:302] T 00000000000000000000000000000000 P 9536cb4469344db0b7d846da1abf37b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9536cb4469344db0b7d846da1abf37b7; no voters:
I20260810 04:10:36.319372 21728 leader_election.cc:288] T 00000000000000000000000000000000 P 9536cb4469344db0b7d846da1abf37b7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 04:10:36.319398 21731 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9536cb4469344db0b7d846da1abf37b7 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:10:36.319595 21731 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9536cb4469344db0b7d846da1abf37b7 [term 1 LEADER]: Becoming Leader. State: Replica: 9536cb4469344db0b7d846da1abf37b7, State: Running, Role: LEADER
I20260810 04:10:36.319664 21728 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9536cb4469344db0b7d846da1abf37b7 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:10:36.319689 21731 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9536cb4469344db0b7d846da1abf37b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9536cb4469344db0b7d846da1abf37b7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38387 } }
I20260810 04:10:36.320135 21733 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9536cb4469344db0b7d846da1abf37b7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9536cb4469344db0b7d846da1abf37b7. Latest consensus state: current_term: 1 leader_uuid: "9536cb4469344db0b7d846da1abf37b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9536cb4469344db0b7d846da1abf37b7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38387 } } }
I20260810 04:10:36.320226 21733 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9536cb4469344db0b7d846da1abf37b7 [sys.catalog]: This master's current role is: LEADER
I20260810 04:10:36.320397 21732 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9536cb4469344db0b7d846da1abf37b7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9536cb4469344db0b7d846da1abf37b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9536cb4469344db0b7d846da1abf37b7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38387 } } }
I20260810 04:10:36.320544 21732 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9536cb4469344db0b7d846da1abf37b7 [sys.catalog]: This master's current role is: LEADER
I20260810 04:10:36.320580 31872 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskws89Io/build/release/bin/kudu as pid 21656
I20260810 04:10:36.320705 31872 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786334794676589-31872-0/minicluster-data/master-0/wal/instance
I20260810 04:10:36.320904 31872 external_mini_cluster.cc:1428] Setting key a5b6a2f46da3348aba1d90cfedf85c3a
I20260810 04:10:36.321404 21739 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 04:10:36.322602 21739 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 04:10:36.324177 21739 catalog_manager.cc:1347] Generated new cluster ID: fb3885e00303498b9043fa32fafaeb15
I20260810 04:10:36.324231 21739 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 10 04:10:36 dist-test-slave-8w9v krb5kdc[21642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335036, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260810 04:10:36.331000 21739 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 04:10:36.331686 21739 catalog_manager.cc:1504] Loading token signing keys...
I20260810 04:10:36.333640 21744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:36.321489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34841 (local address 127.0.0.1:38387)
0810 04:10:36.321689 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0810 04:10:36.321702 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 04:10:36.321949 (+ 247us) server_negotiation.cc:407] Connection header received
0810 04:10:36.322077 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:10:36.322083 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:10:36.322469 (+ 386us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:10:36.323461 (+ 992us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:10:36.323555 (+ 94us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:36.324931 (+ 1376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:36.325985 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:36.326619 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:36.326736 (+ 117us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:10:36.329607 (+ 2871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:10:36.329611 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:10:36.329612 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:10:36.329623 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:10:36.331824 (+ 2201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:10:36.332307 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:10:36.332310 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:10:36.332311 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:10:36.332417 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:10:36.332679 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:10:36.332679 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:10:36.332680 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:10:36.332940 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:10:36.333097 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0810 04:10:36.333227 (+ 130us) server_negotiation.cc:299] Negotiation successful
0810 04:10:36.333427 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":102,"threads_started":1}
I20260810 04:10:36.334419 21739 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9536cb4469344db0b7d846da1abf37b7: Generated new TSK 0
I20260810 04:10:36.334575 21739 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 04:10:36.334700 31872 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 04:10:36.343116 21744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:36.336244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46567 (local address 127.0.0.1:38387)
0810 04:10:36.336307 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0810 04:10:36.336309 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:10:36.336480 (+ 171us) server_negotiation.cc:407] Connection header received
0810 04:10:36.336534 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:10:36.336535 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:10:36.336544 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:10:36.336589 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:10:36.337187 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:36.337718 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:36.338795 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:36.339024 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:36.339105 (+ 81us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:10:36.340504 (+ 1399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:10:36.340505 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:10:36.340506 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:10:36.340512 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:10:36.341896 (+ 1384us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:10:36.342331 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:10:36.342332 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:10:36.342332 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:10:36.342384 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:10:36.342618 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:10:36.342618 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:10:36.342618 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:10:36.342711 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:10:36.342785 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0810 04:10:36.342955 (+ 170us) server_negotiation.cc:299] Negotiation successful
0810 04:10:36.343003 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260810 04:10:36.345230 31872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskws89Io/build/release/bin/kudu with pid 21656
W20260810 04:10:36.352213 21640 connection.cc:535] client connection to 127.0.0.1:38387 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38387): Connection reset by peer (error 104)
2026-08-10T04:10:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1391 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786334794676589-31872-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 04:10:36 dist-test-slave-8w9v krb5kdc[21762](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 04:10:36 dist-test-slave-8w9v krb5kdc[21762](info): set up 2 sockets
Aug 10 04:10:36 dist-test-slave-8w9v krb5kdc[21762](info): commencing operation
krb5kdc: starting...
W20260810 04:10:37.429258 31872 mini_kdc.cc:121] Time spent starting KDC: real 1.063s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786334794676589-31872-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786334794676589-31872-0/krb5kdc/test-user.keytab.
Aug 10 04:10:37 dist-test-slave-8w9v krb5kdc[21762](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335037, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T04:10:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T04:10:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786334794676589-31872-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786334794676589-31872-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 04:10:37.557231 31872 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskws89Io/build/release/bin/kudu
/tmp/dist-test-taskws89Io/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786334794676589-31872-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786334794676589-31872-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786334794676589-31872-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786334794676589-31872-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45131
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786334794676589-31872-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45131
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786334794676589-31872-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786334794676589-31872-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786334794676589-31872-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:10:37.640704 21775 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:10:37.640914 21775 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:10:37.640937 21775 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:10:37.642144 21775 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:10:37.642202 21775 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:10:37.642228 21775 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:10:37.642241 21775 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:10:37.642254 21775 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:10:37.643759 21775 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786334794676589-31872-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786334794676589-31872-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45131
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786334794676589-31872-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45131
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786334794676589-31872-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21775
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786334794676589-31872-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:10:37.644021 21775 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:10:37.644296 21775 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:10:37.649281 21786 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:10:37.649281 21782 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:10:37.649427 21775 server_base.cc:1034] running on GCE node
W20260810 04:10:37.649405 21784 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:10:37.649842 21775 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:10:37.650158 21775 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:10:37.651332 21775 hybrid_clock.cc:648] HybridClock initialized: now 1786335037651320 us; error 47 us; skew 500 ppm
Aug 10 04:10:37 dist-test-slave-8w9v krb5kdc[21762](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335037, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:10:37.653843 21775 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 04:10:37.654327 21775 webserver.cc:460] Webserver started at http://127.0.0.1:40987/ using document root <none> and password file <none>
I20260810 04:10:37.654570 21775 fs_manager.cc:359] Metadata directory not provided
I20260810 04:10:37.654660 21775 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:10:37.654817 21775 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:10:37.655900 21775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786334794676589-31872-0/minicluster-data/master-0/data/instance:
uuid: "11866645754f45e5a6fc2e9b404f4806"
format_stamp: "Formatted at 2026-08-10 04:10:37 on dist-test-slave-8w9v"
server_key: "98a8ad2af280dca663a01d10f690646c"
server_key_iv: "929b59675ba3bf55fc1c498a78feaa39"
server_key_version: "encryptionkey@0"
I20260810 04:10:37.656371 21775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786334794676589-31872-0/minicluster-data/master-0/wal/instance:
uuid: "11866645754f45e5a6fc2e9b404f4806"
format_stamp: "Formatted at 2026-08-10 04:10:37 on dist-test-slave-8w9v"
server_key: "98a8ad2af280dca663a01d10f690646c"
server_key_iv: "929b59675ba3bf55fc1c498a78feaa39"
server_key_version: "encryptionkey@0"
I20260810 04:10:37.657716 21775 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260810 04:10:37.658478 21793 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:10:37.658663 21775 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 04:10:37.658768 21775 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786334794676589-31872-0/minicluster-data/master-0/data,/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786334794676589-31872-0/minicluster-data/master-0/wal
uuid: "11866645754f45e5a6fc2e9b404f4806"
format_stamp: "Formatted at 2026-08-10 04:10:37 on dist-test-slave-8w9v"
server_key: "98a8ad2af280dca663a01d10f690646c"
server_key_iv: "929b59675ba3bf55fc1c498a78feaa39"
server_key_version: "encryptionkey@0"
I20260810 04:10:37.658856 21775 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786334794676589-31872-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786334794676589-31872-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786334794676589-31872-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:10:37.667730 21775 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:10:37.669548 21775 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:10:37.669710 21775 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:10:37.674253 21775 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45131
I20260810 04:10:37.674285 21852 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45131 every 8 connection(s)
I20260810 04:10:37.674660 21775 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786334794676589-31872-0/minicluster-data/master-0/data/info.pb
I20260810 04:10:37.675498 21854 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:10:37.677800 21854 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 11866645754f45e5a6fc2e9b404f4806: Bootstrap starting.
I20260810 04:10:37.678344 21854 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 11866645754f45e5a6fc2e9b404f4806: Neither blocks nor log segments found. Creating new log.
I20260810 04:10:37.678581 21854 log.cc:824] T 00000000000000000000000000000000 P 11866645754f45e5a6fc2e9b404f4806: Log is configured to *not* fsync() on all Append() calls
I20260810 04:10:37.679265 21854 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 11866645754f45e5a6fc2e9b404f4806: No bootstrap required, opened a new log
I20260810 04:10:37.680903 21854 raft_consensus.cc:348] T 00000000000000000000000000000000 P 11866645754f45e5a6fc2e9b404f4806 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11866645754f45e5a6fc2e9b404f4806" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45131 } }
I20260810 04:10:37.681041 21854 raft_consensus.cc:374] T 00000000000000000000000000000000 P 11866645754f45e5a6fc2e9b404f4806 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:10:37.681062 21854 raft_consensus.cc:729] T 00000000000000000000000000000000 P 11866645754f45e5a6fc2e9b404f4806 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11866645754f45e5a6fc2e9b404f4806, State: Initialized, Role: FOLLOWER
I20260810 04:10:37.681211 21854 consensus_queue.cc:260] T 00000000000000000000000000000000 P 11866645754f45e5a6fc2e9b404f4806 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11866645754f45e5a6fc2e9b404f4806" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45131 } }
I20260810 04:10:37.681291 21854 raft_consensus.cc:388] T 00000000000000000000000000000000 P 11866645754f45e5a6fc2e9b404f4806 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 04:10:37.681315 21854 raft_consensus.cc:482] T 00000000000000000000000000000000 P 11866645754f45e5a6fc2e9b404f4806 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 04:10:37.681344 21854 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 11866645754f45e5a6fc2e9b404f4806 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:10:37.681816 21854 raft_consensus.cc:504] T 00000000000000000000000000000000 P 11866645754f45e5a6fc2e9b404f4806 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11866645754f45e5a6fc2e9b404f4806" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45131 } }
I20260810 04:10:37.681903 21854 leader_election.cc:302] T 00000000000000000000000000000000 P 11866645754f45e5a6fc2e9b404f4806 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 11866645754f45e5a6fc2e9b404f4806; no voters:
I20260810 04:10:37.682040 21854 leader_election.cc:288] T 00000000000000000000000000000000 P 11866645754f45e5a6fc2e9b404f4806 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 04:10:37.682111 21857 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 11866645754f45e5a6fc2e9b404f4806 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:10:37.682330 21854 sys_catalog.cc:564] T 00000000000000000000000000000000 P 11866645754f45e5a6fc2e9b404f4806 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:10:37.682329 21857 raft_consensus.cc:686] T 00000000000000000000000000000000 P 11866645754f45e5a6fc2e9b404f4806 [term 1 LEADER]: Becoming Leader. State: Replica: 11866645754f45e5a6fc2e9b404f4806, State: Running, Role: LEADER
I20260810 04:10:37.682545 21857 consensus_queue.cc:237] T 00000000000000000000000000000000 P 11866645754f45e5a6fc2e9b404f4806 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11866645754f45e5a6fc2e9b404f4806" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45131 } }
I20260810 04:10:37.682829 31872 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskws89Io/build/release/bin/kudu as pid 21775
I20260810 04:10:37.682965 31872 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786334794676589-31872-0/minicluster-data/master-0/wal/instance
I20260810 04:10:37.683117 21859 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11866645754f45e5a6fc2e9b404f4806 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 11866645754f45e5a6fc2e9b404f4806. Latest consensus state: current_term: 1 leader_uuid: "11866645754f45e5a6fc2e9b404f4806" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11866645754f45e5a6fc2e9b404f4806" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45131 } } }
I20260810 04:10:37.683187 31872 external_mini_cluster.cc:1428] Setting key b2828700d8aaf68c498a373adcba4e46
I20260810 04:10:37.683205 21859 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11866645754f45e5a6fc2e9b404f4806 [sys.catalog]: This master's current role is: LEADER
I20260810 04:10:37.684087 21858 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11866645754f45e5a6fc2e9b404f4806 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "11866645754f45e5a6fc2e9b404f4806" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11866645754f45e5a6fc2e9b404f4806" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45131 } } }
I20260810 04:10:37.684202 21858 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11866645754f45e5a6fc2e9b404f4806 [sys.catalog]: This master's current role is: LEADER
W20260810 04:10:37.684716 21877 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 11866645754f45e5a6fc2e9b404f4806: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 04:10:37.684790 21877 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 04:10:37.684880 21867 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 04:10:37.685062 21867 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 04:10:37.686532 21867 catalog_manager.cc:1347] Generated new cluster ID: b21bc1598c144209be9dea54ca647acf
I20260810 04:10:37.686584 21867 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 04:10:37.689311 21876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:37.683637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59309 (local address 127.0.0.1:45131)
0810 04:10:37.683921 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0810 04:10:37.683930 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 04:10:37.683968 (+ 38us) server_negotiation.cc:407] Connection header received
0810 04:10:37.684106 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:10:37.684113 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:10:37.684199 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:10:37.684362 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:10:37.684639 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:37.686017 (+ 1378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:37.687124 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:37.687734 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:37.687830 (+ 96us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:10:37.688617 (+ 787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:10:37.688621 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:10:37.688621 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:10:37.688815 (+ 194us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:10:37.688850 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:10:37.688864 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:10:37.688951 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0810 04:10:37.689088 (+ 137us) server_negotiation.cc:299] Negotiation successful
0810 04:10:37.689114 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":230,"thread_start_us":62,"threads_started":1}
I20260810 04:10:37.690507 31872 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 04:10:37.691577 21867 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 04:10:37.692281 21867 catalog_manager.cc:1504] Loading token signing keys...
I20260810 04:10:37.695763 21876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:37.692224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40739 (local address 127.0.0.1:45131)
0810 04:10:37.692258 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0810 04:10:37.692259 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 04:10:37.692306 (+ 47us) server_negotiation.cc:407] Connection header received
0810 04:10:37.692420 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:10:37.692421 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:10:37.692432 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:10:37.692489 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:10:37.692624 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:37.693372 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:37.694444 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:37.694741 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:37.694798 (+ 57us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:10:37.695515 (+ 717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:10:37.695516 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:10:37.695516 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:10:37.695520 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:10:37.695536 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:10:37.695540 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:10:37.695598 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0810 04:10:37.695684 (+ 86us) server_negotiation.cc:299] Negotiation successful
0810 04:10:37.695695 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260810 04:10:37.697613 21867 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 11866645754f45e5a6fc2e9b404f4806: Generated new TSK 0
I20260810 04:10:37.697772 21867 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 04:10:37.703330 31872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskws89Io/build/release/bin/kudu with pid 21775
W20260810 04:10:37.715077 21759 connection.cc:535] client connection to 127.0.0.1:45131 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45131): Connection reset by peer (error 104)
2026-08-10T04:10:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1362 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786334794676589-31872-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 04:10:37 dist-test-slave-8w9v krb5kdc[21887](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 04:10:37 dist-test-slave-8w9v krb5kdc[21887](info): set up 2 sockets
Aug 10 04:10:37 dist-test-slave-8w9v krb5kdc[21887](info): commencing operation
krb5kdc: starting...
W20260810 04:10:38.807190 31872 mini_kdc.cc:121] Time spent starting KDC: real 1.079s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786334794676589-31872-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786334794676589-31872-0/krb5kdc/test-user.keytab.
Aug 10 04:10:38 dist-test-slave-8w9v krb5kdc[21887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335038, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T04:10:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T04:10:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786334794676589-31872-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786334794676589-31872-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 04:10:38.935526 31872 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskws89Io/build/release/bin/kudu
/tmp/dist-test-taskws89Io/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786334794676589-31872-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786334794676589-31872-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786334794676589-31872-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786334794676589-31872-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35939
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786334794676589-31872-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35939
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786334794676589-31872-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786334794676589-31872-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786334794676589-31872-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:10:39.019135 21902 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:10:39.019400 21902 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:10:39.019444 21902 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:10:39.020576 21902 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:10:39.020655 21902 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:10:39.020679 21902 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:10:39.020707 21902 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:10:39.020735 21902 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:10:39.022131 21902 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786334794676589-31872-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786334794676589-31872-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35939
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786334794676589-31872-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35939
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786334794676589-31872-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21902
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786334794676589-31872-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:10:39.022432 21902 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:10:39.022665 21902 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:10:39.027590 21908 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:10:39.027606 21911 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:10:39.027608 21909 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:10:39.027920 21902 server_base.cc:1034] running on GCE node
I20260810 04:10:39.028266 21902 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:10:39.028595 21902 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:10:39.029815 21902 hybrid_clock.cc:648] HybridClock initialized: now 1786335039029794 us; error 30 us; skew 500 ppm
Aug 10 04:10:39 dist-test-slave-8w9v krb5kdc[21887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335039, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:10:39.037068 21902 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 04:10:39.037667 21902 webserver.cc:460] Webserver started at http://127.0.0.1:39735/ using document root <none> and password file <none>
I20260810 04:10:39.037935 21902 fs_manager.cc:359] Metadata directory not provided
I20260810 04:10:39.038019 21902 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:10:39.038194 21902 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:10:39.039436 21902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786334794676589-31872-0/minicluster-data/master-0/data/instance:
uuid: "54098388916d4cbcb4472a00da59867c"
format_stamp: "Formatted at 2026-08-10 04:10:39 on dist-test-slave-8w9v"
server_key: "e27132e143fd5fa11aa91ee714adf503"
server_key_iv: "0cdcbf1b38919b3c753da731b2545e0f"
server_key_version: "encryptionkey@0"
I20260810 04:10:39.039876 21902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786334794676589-31872-0/minicluster-data/master-0/wal/instance:
uuid: "54098388916d4cbcb4472a00da59867c"
format_stamp: "Formatted at 2026-08-10 04:10:39 on dist-test-slave-8w9v"
server_key: "e27132e143fd5fa11aa91ee714adf503"
server_key_iv: "0cdcbf1b38919b3c753da731b2545e0f"
server_key_version: "encryptionkey@0"
I20260810 04:10:39.041463 21902 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.004s sys 0.000s
I20260810 04:10:39.042351 21919 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:10:39.042562 21902 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:10:39.042685 21902 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786334794676589-31872-0/minicluster-data/master-0/data,/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786334794676589-31872-0/minicluster-data/master-0/wal
uuid: "54098388916d4cbcb4472a00da59867c"
format_stamp: "Formatted at 2026-08-10 04:10:39 on dist-test-slave-8w9v"
server_key: "e27132e143fd5fa11aa91ee714adf503"
server_key_iv: "0cdcbf1b38919b3c753da731b2545e0f"
server_key_version: "encryptionkey@0"
I20260810 04:10:39.042773 21902 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786334794676589-31872-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786334794676589-31872-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786334794676589-31872-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:10:39.046615 21902 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:10:39.048720 21902 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:10:39.048903 21902 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:10:39.053656 21902 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35939
I20260810 04:10:39.053705 21977 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35939 every 8 connection(s)
I20260810 04:10:39.054093 21902 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786334794676589-31872-0/minicluster-data/master-0/data/info.pb
I20260810 04:10:39.055143 21978 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:10:39.057608 21978 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54098388916d4cbcb4472a00da59867c: Bootstrap starting.
I20260810 04:10:39.058195 21978 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 54098388916d4cbcb4472a00da59867c: Neither blocks nor log segments found. Creating new log.
I20260810 04:10:39.058506 21978 log.cc:824] T 00000000000000000000000000000000 P 54098388916d4cbcb4472a00da59867c: Log is configured to *not* fsync() on all Append() calls
I20260810 04:10:39.059232 21978 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54098388916d4cbcb4472a00da59867c: No bootstrap required, opened a new log
I20260810 04:10:39.060665 31872 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskws89Io/build/release/bin/kudu as pid 21902
I20260810 04:10:39.060763 31872 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786334794676589-31872-0/minicluster-data/master-0/wal/instance
I20260810 04:10:39.060967 31872 external_mini_cluster.cc:1428] Setting key c85b18cb69d7758b308334cd3e87df29
I20260810 04:10:39.061226 21978 raft_consensus.cc:348] T 00000000000000000000000000000000 P 54098388916d4cbcb4472a00da59867c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54098388916d4cbcb4472a00da59867c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35939 } }
I20260810 04:10:39.061424 21978 raft_consensus.cc:374] T 00000000000000000000000000000000 P 54098388916d4cbcb4472a00da59867c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:10:39.061465 21978 raft_consensus.cc:729] T 00000000000000000000000000000000 P 54098388916d4cbcb4472a00da59867c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54098388916d4cbcb4472a00da59867c, State: Initialized, Role: FOLLOWER
I20260810 04:10:39.061651 21978 consensus_queue.cc:260] T 00000000000000000000000000000000 P 54098388916d4cbcb4472a00da59867c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54098388916d4cbcb4472a00da59867c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35939 } }
I20260810 04:10:39.061749 21978 raft_consensus.cc:388] T 00000000000000000000000000000000 P 54098388916d4cbcb4472a00da59867c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 04:10:39.061776 21978 raft_consensus.cc:482] T 00000000000000000000000000000000 P 54098388916d4cbcb4472a00da59867c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 04:10:39.061830 21978 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 54098388916d4cbcb4472a00da59867c [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:10:39.062903 21978 raft_consensus.cc:504] T 00000000000000000000000000000000 P 54098388916d4cbcb4472a00da59867c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54098388916d4cbcb4472a00da59867c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35939 } }
I20260810 04:10:39.062994 21978 leader_election.cc:302] T 00000000000000000000000000000000 P 54098388916d4cbcb4472a00da59867c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 54098388916d4cbcb4472a00da59867c; no voters:
I20260810 04:10:39.063218 21978 leader_election.cc:288] T 00000000000000000000000000000000 P 54098388916d4cbcb4472a00da59867c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 04:10:39.063347 21983 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 54098388916d4cbcb4472a00da59867c [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:10:39.063493 21978 sys_catalog.cc:564] T 00000000000000000000000000000000 P 54098388916d4cbcb4472a00da59867c [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:10:39.063552 21983 raft_consensus.cc:686] T 00000000000000000000000000000000 P 54098388916d4cbcb4472a00da59867c [term 1 LEADER]: Becoming Leader. State: Replica: 54098388916d4cbcb4472a00da59867c, State: Running, Role: LEADER
I20260810 04:10:39.063552 21982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:39.061508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40923 (local address 127.0.0.1:35939)
0810 04:10:39.061700 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0810 04:10:39.061710 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 04:10:39.061745 (+ 35us) server_negotiation.cc:407] Connection header received
0810 04:10:39.061893 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:10:39.061901 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:10:39.061932 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:10:39.062074 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:10:39.062888 (+ 814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:10:39.062891 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:10:39.062892 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:10:39.062896 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:10:39.062950 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:10:39.062985 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:10:39.063049 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0810 04:10:39.063218 (+ 169us) server_negotiation.cc:299] Negotiation successful
0810 04:10:39.063266 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":96,"threads_started":1}
I20260810 04:10:39.063683 21983 consensus_queue.cc:237] T 00000000000000000000000000000000 P 54098388916d4cbcb4472a00da59867c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54098388916d4cbcb4472a00da59867c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35939 } }
I20260810 04:10:39.064265 21984 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54098388916d4cbcb4472a00da59867c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "54098388916d4cbcb4472a00da59867c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54098388916d4cbcb4472a00da59867c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35939 } } }
I20260810 04:10:39.064361 21984 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54098388916d4cbcb4472a00da59867c [sys.catalog]: This master's current role is: LEADER
I20260810 04:10:39.064563 21985 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54098388916d4cbcb4472a00da59867c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 54098388916d4cbcb4472a00da59867c. Latest consensus state: current_term: 1 leader_uuid: "54098388916d4cbcb4472a00da59867c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54098388916d4cbcb4472a00da59867c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35939 } } }
I20260810 04:10:39.064620 21985 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54098388916d4cbcb4472a00da59867c [sys.catalog]: This master's current role is: LEADER
I20260810 04:10:39.064620 21993 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 04:10:39.065199 21993 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 04:10:39.065477 31872 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 04:10:39.067394 21993 catalog_manager.cc:1347] Generated new cluster ID: 804d3e477828432dbd7966b407870969
I20260810 04:10:39.067443 21993 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 04:10:39.069108 21982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:39.066950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42765 (local address 127.0.0.1:35939)
0810 04:10:39.066988 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0810 04:10:39.066990 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:10:39.067298 (+ 308us) server_negotiation.cc:407] Connection header received
0810 04:10:39.067323 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:10:39.067324 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:10:39.067329 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:10:39.067397 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:10:39.068881 (+ 1484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:10:39.068882 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:10:39.068883 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:10:39.068887 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:10:39.068901 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:10:39.068905 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:10:39.068977 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0810 04:10:39.069028 (+ 51us) server_negotiation.cc:299] Negotiation successful
0810 04:10:39.069043 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260810 04:10:39.098542 21993 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 04:10:39.098743 21993 catalog_manager.cc:1504] Loading token signing keys...
I20260810 04:10:39.104279 21993 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 54098388916d4cbcb4472a00da59867c: Generated new TSK 0
I20260810 04:10:39.104503 21993 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 04:10:39.109370 31872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskws89Io/build/release/bin/kudu with pid 21902
2026-08-10T04:10:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1401 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786334794676589-31872-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 04:10:39 dist-test-slave-8w9v krb5kdc[22012](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 04:10:39 dist-test-slave-8w9v krb5kdc[22012](info): set up 2 sockets
Aug 10 04:10:39 dist-test-slave-8w9v krb5kdc[22012](info): commencing operation
krb5kdc: starting...
W20260810 04:10:40.204258 31872 mini_kdc.cc:121] Time spent starting KDC: real 1.074s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786334794676589-31872-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786334794676589-31872-0/krb5kdc/test-user.keytab.
Aug 10 04:10:40 dist-test-slave-8w9v krb5kdc[22012](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335040, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T04:10:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T04:10:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786334794676589-31872-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786334794676589-31872-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 04:10:40.331089 31872 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskws89Io/build/release/bin/kudu
/tmp/dist-test-taskws89Io/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786334794676589-31872-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786334794676589-31872-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786334794676589-31872-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786334794676589-31872-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38121
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786334794676589-31872-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38121
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786334794676589-31872-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786334794676589-31872-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786334794676589-31872-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:10:40.414304 22026 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:10:40.414547 22026 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:10:40.414589 22026 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:10:40.415733 22026 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:10:40.415812 22026 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:10:40.415836 22026 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:10:40.415864 22026 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:10:40.415894 22026 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:10:40.417404 22026 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786334794676589-31872-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786334794676589-31872-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38121
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786334794676589-31872-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38121
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786334794676589-31872-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22026
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786334794676589-31872-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:10:40.417683 22026 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:10:40.417917 22026 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:10:40.423074 22031 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:10:40.423058 22034 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:10:40.423050 22032 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:10:40.423250 22026 server_base.cc:1034] running on GCE node
I20260810 04:10:40.423564 22026 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:10:40.423872 22026 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:10:40.425020 22026 hybrid_clock.cc:648] HybridClock initialized: now 1786335040425011 us; error 31 us; skew 500 ppm
Aug 10 04:10:40 dist-test-slave-8w9v krb5kdc[22012](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335040, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:10:40.427599 22026 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 04:10:40.427974 22026 webserver.cc:460] Webserver started at http://127.0.0.1:39201/ using document root <none> and password file <none>
I20260810 04:10:40.428229 22026 fs_manager.cc:359] Metadata directory not provided
I20260810 04:10:40.428280 22026 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:10:40.428378 22026 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:10:40.429418 22026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786334794676589-31872-0/minicluster-data/master-0/data/instance:
uuid: "2dfafcdb911d49fea8b87ca4b5306b13"
format_stamp: "Formatted at 2026-08-10 04:10:40 on dist-test-slave-8w9v"
server_key: "56c7abb1287dc6c668842b6ba970a324"
server_key_iv: "d6eb9167bbf175836945cdc8d9ebcfdb"
server_key_version: "encryptionkey@0"
I20260810 04:10:40.429723 22026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786334794676589-31872-0/minicluster-data/master-0/wal/instance:
uuid: "2dfafcdb911d49fea8b87ca4b5306b13"
format_stamp: "Formatted at 2026-08-10 04:10:40 on dist-test-slave-8w9v"
server_key: "56c7abb1287dc6c668842b6ba970a324"
server_key_iv: "d6eb9167bbf175836945cdc8d9ebcfdb"
server_key_version: "encryptionkey@0"
I20260810 04:10:40.431099 22026 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 04:10:40.431784 22042 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:10:40.431968 22026 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 04:10:40.432042 22026 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786334794676589-31872-0/minicluster-data/master-0/data,/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786334794676589-31872-0/minicluster-data/master-0/wal
uuid: "2dfafcdb911d49fea8b87ca4b5306b13"
format_stamp: "Formatted at 2026-08-10 04:10:40 on dist-test-slave-8w9v"
server_key: "56c7abb1287dc6c668842b6ba970a324"
server_key_iv: "d6eb9167bbf175836945cdc8d9ebcfdb"
server_key_version: "encryptionkey@0"
I20260810 04:10:40.432156 22026 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786334794676589-31872-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786334794676589-31872-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786334794676589-31872-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:10:40.443693 22026 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:10:40.446172 22026 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:10:40.446341 22026 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:10:40.450763 22026 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38121
I20260810 04:10:40.450799 22099 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38121 every 8 connection(s)
I20260810 04:10:40.451171 22026 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786334794676589-31872-0/minicluster-data/master-0/data/info.pb
I20260810 04:10:40.452062 22100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:10:40.454392 22100 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2dfafcdb911d49fea8b87ca4b5306b13: Bootstrap starting.
I20260810 04:10:40.454977 22100 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2dfafcdb911d49fea8b87ca4b5306b13: Neither blocks nor log segments found. Creating new log.
I20260810 04:10:40.455271 22100 log.cc:824] T 00000000000000000000000000000000 P 2dfafcdb911d49fea8b87ca4b5306b13: Log is configured to *not* fsync() on all Append() calls
I20260810 04:10:40.455974 22100 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2dfafcdb911d49fea8b87ca4b5306b13: No bootstrap required, opened a new log
I20260810 04:10:40.456056 31872 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskws89Io/build/release/bin/kudu as pid 22026
I20260810 04:10:40.456198 31872 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786334794676589-31872-0/minicluster-data/master-0/wal/instance
I20260810 04:10:40.456399 31872 external_mini_cluster.cc:1428] Setting key 7ced819b0257ecec42ae0141835a890e
I20260810 04:10:40.458307 22100 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2dfafcdb911d49fea8b87ca4b5306b13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dfafcdb911d49fea8b87ca4b5306b13" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38121 } }
I20260810 04:10:40.458519 22100 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2dfafcdb911d49fea8b87ca4b5306b13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:10:40.458555 22100 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2dfafcdb911d49fea8b87ca4b5306b13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2dfafcdb911d49fea8b87ca4b5306b13, State: Initialized, Role: FOLLOWER
I20260810 04:10:40.458670 22100 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2dfafcdb911d49fea8b87ca4b5306b13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dfafcdb911d49fea8b87ca4b5306b13" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38121 } }
I20260810 04:10:40.458757 22100 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2dfafcdb911d49fea8b87ca4b5306b13 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 04:10:40.458792 22100 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2dfafcdb911d49fea8b87ca4b5306b13 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 04:10:40.458882 22100 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2dfafcdb911d49fea8b87ca4b5306b13 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:10:40.459486 22100 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2dfafcdb911d49fea8b87ca4b5306b13 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dfafcdb911d49fea8b87ca4b5306b13" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38121 } }
I20260810 04:10:40.459592 22100 leader_election.cc:302] T 00000000000000000000000000000000 P 2dfafcdb911d49fea8b87ca4b5306b13 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2dfafcdb911d49fea8b87ca4b5306b13; no voters:
I20260810 04:10:40.459735 22100 leader_election.cc:288] T 00000000000000000000000000000000 P 2dfafcdb911d49fea8b87ca4b5306b13 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 04:10:40.459787 22105 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2dfafcdb911d49fea8b87ca4b5306b13 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:10:40.460007 22105 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2dfafcdb911d49fea8b87ca4b5306b13 [term 1 LEADER]: Becoming Leader. State: Replica: 2dfafcdb911d49fea8b87ca4b5306b13, State: Running, Role: LEADER
I20260810 04:10:40.459990 22100 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2dfafcdb911d49fea8b87ca4b5306b13 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:10:40.460192 22105 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2dfafcdb911d49fea8b87ca4b5306b13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dfafcdb911d49fea8b87ca4b5306b13" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38121 } }
I20260810 04:10:40.461161 22106 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2dfafcdb911d49fea8b87ca4b5306b13 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2dfafcdb911d49fea8b87ca4b5306b13" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dfafcdb911d49fea8b87ca4b5306b13" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38121 } } }
I20260810 04:10:40.461258 22106 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2dfafcdb911d49fea8b87ca4b5306b13 [sys.catalog]: This master's current role is: LEADER
I20260810 04:10:40.461607 22107 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2dfafcdb911d49fea8b87ca4b5306b13 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2dfafcdb911d49fea8b87ca4b5306b13. Latest consensus state: current_term: 1 leader_uuid: "2dfafcdb911d49fea8b87ca4b5306b13" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dfafcdb911d49fea8b87ca4b5306b13" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38121 } } }
W20260810 04:10:40.462095 22123 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2dfafcdb911d49fea8b87ca4b5306b13: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260810 04:10:40.462100 22107 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2dfafcdb911d49fea8b87ca4b5306b13 [sys.catalog]: This master's current role is: LEADER
W20260810 04:10:40.462183 22123 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 04:10:40.462318 22119 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 04:10:40.462489 22119 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 04:10:40.464290 22119 catalog_manager.cc:1347] Generated new cluster ID: 93b55249601f417fbdffdb56ad90b85b
I20260810 04:10:40.464345 22119 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 10 04:10:40 dist-test-slave-8w9v krb5kdc[22012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335040, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260810 04:10:40.468629 22104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:40.456936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45421 (local address 127.0.0.1:38121)
0810 04:10:40.457139 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0810 04:10:40.457148 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 04:10:40.457182 (+ 34us) server_negotiation.cc:407] Connection header received
0810 04:10:40.458524 (+ 1342us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:10:40.458533 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:10:40.458596 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:10:40.458749 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:10:40.459465 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:40.460924 (+ 1459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:40.461977 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:40.462613 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:40.462717 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:10:40.465369 (+ 2652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:10:40.465373 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:10:40.465373 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:10:40.465385 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:10:40.467068 (+ 1683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:10:40.467456 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:10:40.467459 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:10:40.467459 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:10:40.467549 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:10:40.467791 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:10:40.467792 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:10:40.467792 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:10:40.468023 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:10:40.468218 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0810 04:10:40.468331 (+ 113us) server_negotiation.cc:299] Negotiation successful
0810 04:10:40.468483 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":94,"threads_started":1}
I20260810 04:10:40.469540 31872 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 04:10:40.478318 22104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:40.470990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39419 (local address 127.0.0.1:38121)
0810 04:10:40.471027 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0810 04:10:40.471029 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:10:40.472372 (+ 1343us) server_negotiation.cc:407] Connection header received
0810 04:10:40.472446 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:10:40.472447 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:10:40.472457 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:10:40.472512 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:10:40.473028 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:40.473492 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:40.474191 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:40.474417 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:40.474499 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:10:40.475682 (+ 1183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:10:40.475683 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:10:40.475684 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:10:40.475690 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:10:40.477149 (+ 1459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:10:40.477529 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:10:40.477531 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:10:40.477531 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:10:40.477586 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:10:40.477802 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:10:40.477803 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:10:40.477803 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:10:40.477924 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:10:40.478026 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0810 04:10:40.478136 (+ 110us) server_negotiation.cc:299] Negotiation successful
0810 04:10:40.478211 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260810 04:10:40.481601 22119 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 04:10:40.482132 22119 catalog_manager.cc:1504] Loading token signing keys...
I20260810 04:10:40.486598 22119 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2dfafcdb911d49fea8b87ca4b5306b13: Generated new TSK 0
I20260810 04:10:40.486781 22119 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 04:10:40.492650 31872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskws89Io/build/release/bin/kudu with pid 22026
2026-08-10T04:10:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1385 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786334794676589-31872-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 04:10:40 dist-test-slave-8w9v krb5kdc[22135](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 04:10:40 dist-test-slave-8w9v krb5kdc[22135](info): set up 2 sockets
Aug 10 04:10:40 dist-test-slave-8w9v krb5kdc[22135](info): commencing operation
krb5kdc: starting...
W20260810 04:10:41.607728 31872 mini_kdc.cc:121] Time spent starting KDC: real 1.092s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786334794676589-31872-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786334794676589-31872-0/krb5kdc/test-user.keytab.
Aug 10 04:10:41 dist-test-slave-8w9v krb5kdc[22135](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335041, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T04:10:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T04:10:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786334794676589-31872-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786334794676589-31872-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 04:10:41.731782 31872 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskws89Io/build/release/bin/kudu
/tmp/dist-test-taskws89Io/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786334794676589-31872-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786334794676589-31872-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786334794676589-31872-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786334794676589-31872-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37539
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786334794676589-31872-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37539
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786334794676589-31872-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786334794676589-31872-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786334794676589-31872-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:10:41.815344 22148 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:10:41.815588 22148 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:10:41.815629 22148 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:10:41.816876 22148 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:10:41.816946 22148 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:10:41.816972 22148 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:10:41.817004 22148 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:10:41.817024 22148 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:10:41.818454 22148 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786334794676589-31872-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786334794676589-31872-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37539
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786334794676589-31872-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37539
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786334794676589-31872-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22148
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786334794676589-31872-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:10:41.818718 22148 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:10:41.818938 22148 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 04:10:41.823949 22148 server_base.cc:1034] running on GCE node
W20260810 04:10:41.823928 22153 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:10:41.823928 22155 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:10:41.823923 22157 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:10:41.824422 22148 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:10:41.824728 22148 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:10:41.825915 22148 hybrid_clock.cc:648] HybridClock initialized: now 1786335041825906 us; error 40 us; skew 500 ppm
Aug 10 04:10:41 dist-test-slave-8w9v krb5kdc[22135](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335041, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:10:41.828622 22148 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 04:10:41.829035 22148 webserver.cc:460] Webserver started at http://127.0.0.1:36243/ using document root <none> and password file <none>
I20260810 04:10:41.829277 22148 fs_manager.cc:359] Metadata directory not provided
I20260810 04:10:41.829347 22148 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:10:41.829506 22148 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:10:41.830560 22148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786334794676589-31872-0/minicluster-data/master-0/data/instance:
uuid: "d022a017840a48a4945bb295282aced2"
format_stamp: "Formatted at 2026-08-10 04:10:41 on dist-test-slave-8w9v"
server_key: "d348f0f83d878b1b41b80a817a91649c"
server_key_iv: "8f3c056833c7cfdf69fa1d8266ea9bf3"
server_key_version: "encryptionkey@0"
I20260810 04:10:41.830932 22148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786334794676589-31872-0/minicluster-data/master-0/wal/instance:
uuid: "d022a017840a48a4945bb295282aced2"
format_stamp: "Formatted at 2026-08-10 04:10:41 on dist-test-slave-8w9v"
server_key: "d348f0f83d878b1b41b80a817a91649c"
server_key_iv: "8f3c056833c7cfdf69fa1d8266ea9bf3"
server_key_version: "encryptionkey@0"
I20260810 04:10:41.832355 22148 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260810 04:10:41.833137 22164 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:10:41.833312 22148 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 04:10:41.833411 22148 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786334794676589-31872-0/minicluster-data/master-0/data,/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786334794676589-31872-0/minicluster-data/master-0/wal
uuid: "d022a017840a48a4945bb295282aced2"
format_stamp: "Formatted at 2026-08-10 04:10:41 on dist-test-slave-8w9v"
server_key: "d348f0f83d878b1b41b80a817a91649c"
server_key_iv: "8f3c056833c7cfdf69fa1d8266ea9bf3"
server_key_version: "encryptionkey@0"
I20260810 04:10:41.833501 22148 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786334794676589-31872-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786334794676589-31872-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786334794676589-31872-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:10:41.855883 22148 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:10:41.858371 22148 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:10:41.858560 22148 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:10:41.862686 22148 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37539
I20260810 04:10:41.862727 22228 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37539 every 8 connection(s)
I20260810 04:10:41.863080 22148 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786334794676589-31872-0/minicluster-data/master-0/data/info.pb
I20260810 04:10:41.863969 22229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:10:41.866276 22229 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d022a017840a48a4945bb295282aced2: Bootstrap starting.
I20260810 04:10:41.866863 22229 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d022a017840a48a4945bb295282aced2: Neither blocks nor log segments found. Creating new log.
I20260810 04:10:41.867005 31872 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskws89Io/build/release/bin/kudu as pid 22148
I20260810 04:10:41.867161 31872 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786334794676589-31872-0/minicluster-data/master-0/wal/instance
I20260810 04:10:41.867261 22229 log.cc:824] T 00000000000000000000000000000000 P d022a017840a48a4945bb295282aced2: Log is configured to *not* fsync() on all Append() calls
I20260810 04:10:41.867373 31872 external_mini_cluster.cc:1428] Setting key f962dad217ada1316b9220ab50bb4eb6
I20260810 04:10:41.868165 22229 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d022a017840a48a4945bb295282aced2: No bootstrap required, opened a new log
I20260810 04:10:41.870620 22229 raft_consensus.cc:348] T 00000000000000000000000000000000 P d022a017840a48a4945bb295282aced2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d022a017840a48a4945bb295282aced2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37539 } }
I20260810 04:10:41.870795 22229 raft_consensus.cc:374] T 00000000000000000000000000000000 P d022a017840a48a4945bb295282aced2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:10:41.870827 22229 raft_consensus.cc:729] T 00000000000000000000000000000000 P d022a017840a48a4945bb295282aced2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d022a017840a48a4945bb295282aced2, State: Initialized, Role: FOLLOWER
I20260810 04:10:41.870949 22229 consensus_queue.cc:260] T 00000000000000000000000000000000 P d022a017840a48a4945bb295282aced2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d022a017840a48a4945bb295282aced2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37539 } }
I20260810 04:10:41.871032 22229 raft_consensus.cc:388] T 00000000000000000000000000000000 P d022a017840a48a4945bb295282aced2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 04:10:41.871070 22229 raft_consensus.cc:482] T 00000000000000000000000000000000 P d022a017840a48a4945bb295282aced2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 04:10:41.871117 22229 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d022a017840a48a4945bb295282aced2 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:10:41.871729 22229 raft_consensus.cc:504] T 00000000000000000000000000000000 P d022a017840a48a4945bb295282aced2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d022a017840a48a4945bb295282aced2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37539 } }
I20260810 04:10:41.871865 22229 leader_election.cc:302] T 00000000000000000000000000000000 P d022a017840a48a4945bb295282aced2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d022a017840a48a4945bb295282aced2; no voters:
I20260810 04:10:41.872354 22229 leader_election.cc:288] T 00000000000000000000000000000000 P d022a017840a48a4945bb295282aced2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 04:10:41.872380 22234 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d022a017840a48a4945bb295282aced2 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:10:41.872577 22234 raft_consensus.cc:686] T 00000000000000000000000000000000 P d022a017840a48a4945bb295282aced2 [term 1 LEADER]: Becoming Leader. State: Replica: d022a017840a48a4945bb295282aced2, State: Running, Role: LEADER
I20260810 04:10:41.872625 22229 sys_catalog.cc:564] T 00000000000000000000000000000000 P d022a017840a48a4945bb295282aced2 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:10:41.872697 22234 consensus_queue.cc:237] T 00000000000000000000000000000000 P d022a017840a48a4945bb295282aced2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d022a017840a48a4945bb295282aced2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37539 } }
I20260810 04:10:41.873020 22234 sys_catalog.cc:455] T 00000000000000000000000000000000 P d022a017840a48a4945bb295282aced2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d022a017840a48a4945bb295282aced2. Latest consensus state: current_term: 1 leader_uuid: "d022a017840a48a4945bb295282aced2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d022a017840a48a4945bb295282aced2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37539 } } }
I20260810 04:10:41.873116 22234 sys_catalog.cc:458] T 00000000000000000000000000000000 P d022a017840a48a4945bb295282aced2 [sys.catalog]: This master's current role is: LEADER
I20260810 04:10:41.873030 22236 sys_catalog.cc:455] T 00000000000000000000000000000000 P d022a017840a48a4945bb295282aced2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d022a017840a48a4945bb295282aced2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d022a017840a48a4945bb295282aced2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37539 } } }
I20260810 04:10:41.873320 22233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:41.867940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49773 (local address 127.0.0.1:37539)
0810 04:10:41.868102 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0810 04:10:41.868112 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 04:10:41.868164 (+ 52us) server_negotiation.cc:407] Connection header received
0810 04:10:41.868649 (+ 485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:10:41.868654 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:10:41.868699 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:10:41.868809 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:10:41.869056 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:41.869922 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:41.871065 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:41.871565 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:41.871646 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:10:41.872364 (+ 718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:10:41.872368 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:10:41.872369 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:10:41.872603 (+ 234us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:10:41.872640 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:10:41.872655 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:10:41.872717 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0810 04:10:41.872921 (+ 204us) server_negotiation.cc:299] Negotiation successful
0810 04:10:41.872984 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":83,"threads_started":1}
I20260810 04:10:41.873327 22236 sys_catalog.cc:458] T 00000000000000000000000000000000 P d022a017840a48a4945bb295282aced2 [sys.catalog]: This master's current role is: LEADER
I20260810 04:10:41.874352 31872 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260810 04:10:41.874594 22250 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d022a017840a48a4945bb295282aced2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 04:10:41.874643 22250 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 04:10:41.874693 22243 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 04:10:41.874887 22243 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 04:10:41.876544 22243 catalog_manager.cc:1347] Generated new cluster ID: 8fbc8e0378c4426d9fbab8553645bf8f
I20260810 04:10:41.876598 22243 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 04:10:41.883584 22233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:41.875942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48119 (local address 127.0.0.1:37539)
0810 04:10:41.875976 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0810 04:10:41.875977 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 04:10:41.880354 (+ 4377us) server_negotiation.cc:407] Connection header received
0810 04:10:41.880382 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:10:41.880383 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:10:41.880394 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:10:41.880468 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:10:41.880599 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:41.881331 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:41.882282 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:41.882605 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:41.882683 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:10:41.883346 (+ 663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:10:41.883347 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:10:41.883348 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:10:41.883352 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:10:41.883368 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:10:41.883372 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:10:41.883433 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0810 04:10:41.883486 (+ 53us) server_negotiation.cc:299] Negotiation successful
0810 04:10:41.883508 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260810 04:10:41.894088 22243 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 04:10:41.894797 22243 catalog_manager.cc:1504] Loading token signing keys...
I20260810 04:10:41.898005 22243 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d022a017840a48a4945bb295282aced2: Generated new TSK 0
I20260810 04:10:41.898181 22243 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 04:10:41.904557 31872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskws89Io/build/release/bin/kudu with pid 22148
2026-08-10T04:10:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1409 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786334794676589-31872-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 04:10:41 dist-test-slave-8w9v krb5kdc[22261](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 04:10:41 dist-test-slave-8w9v krb5kdc[22261](info): set up 2 sockets
Aug 10 04:10:41 dist-test-slave-8w9v krb5kdc[22261](info): commencing operation
krb5kdc: starting...
W20260810 04:10:43.001881 31872 mini_kdc.cc:121] Time spent starting KDC: real 1.076s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786334794676589-31872-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786334794676589-31872-0/krb5kdc/test-user.keytab.
Aug 10 04:10:43 dist-test-slave-8w9v krb5kdc[22261](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335043, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T04:10:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T04:10:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786334794676589-31872-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786334794676589-31872-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 04:10:43.131716 31872 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskws89Io/build/release/bin/kudu
/tmp/dist-test-taskws89Io/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786334794676589-31872-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786334794676589-31872-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786334794676589-31872-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786334794676589-31872-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34017
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786334794676589-31872-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34017
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786334794676589-31872-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786334794676589-31872-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786334794676589-31872-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:10:43.217823 22275 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:10:43.218084 22275 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:10:43.218127 22275 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:10:43.219307 22275 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:10:43.219393 22275 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:10:43.219424 22275 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:10:43.219460 22275 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:10:43.219491 22275 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:10:43.221094 22275 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786334794676589-31872-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786334794676589-31872-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34017
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786334794676589-31872-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34017
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786334794676589-31872-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22275
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786334794676589-31872-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:10:43.221414 22275 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:10:43.221683 22275 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:10:43.226953 22281 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:10:43.226938 22282 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:10:43.226938 22284 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:10:43.227152 22275 server_base.cc:1034] running on GCE node
I20260810 04:10:43.227454 22275 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:10:43.227763 22275 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:10:43.228940 22275 hybrid_clock.cc:648] HybridClock initialized: now 1786335043228934 us; error 38 us; skew 500 ppm
Aug 10 04:10:43 dist-test-slave-8w9v krb5kdc[22261](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335043, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:10:43.234952 22275 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 04:10:43.235461 22275 webserver.cc:460] Webserver started at http://127.0.0.1:34929/ using document root <none> and password file <none>
I20260810 04:10:43.235715 22275 fs_manager.cc:359] Metadata directory not provided
I20260810 04:10:43.235785 22275 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:10:43.235935 22275 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:10:43.237188 22275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786334794676589-31872-0/minicluster-data/master-0/data/instance:
uuid: "ed6d09f42a864c8e92c15ff52d3498a9"
format_stamp: "Formatted at 2026-08-10 04:10:43 on dist-test-slave-8w9v"
server_key: "4cbce5553e3e092203e8d5318e36c216"
server_key_iv: "326ec6ceb29cf0527959a7c9204fedd4"
server_key_version: "encryptionkey@0"
I20260810 04:10:43.237586 22275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786334794676589-31872-0/minicluster-data/master-0/wal/instance:
uuid: "ed6d09f42a864c8e92c15ff52d3498a9"
format_stamp: "Formatted at 2026-08-10 04:10:43 on dist-test-slave-8w9v"
server_key: "4cbce5553e3e092203e8d5318e36c216"
server_key_iv: "326ec6ceb29cf0527959a7c9204fedd4"
server_key_version: "encryptionkey@0"
I20260810 04:10:43.239007 22275 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 04:10:43.239864 22292 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:10:43.240054 22275 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:10:43.240211 22275 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786334794676589-31872-0/minicluster-data/master-0/data,/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786334794676589-31872-0/minicluster-data/master-0/wal
uuid: "ed6d09f42a864c8e92c15ff52d3498a9"
format_stamp: "Formatted at 2026-08-10 04:10:43 on dist-test-slave-8w9v"
server_key: "4cbce5553e3e092203e8d5318e36c216"
server_key_iv: "326ec6ceb29cf0527959a7c9204fedd4"
server_key_version: "encryptionkey@0"
I20260810 04:10:43.240298 22275 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786334794676589-31872-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786334794676589-31872-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786334794676589-31872-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:10:43.243701 22275 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:10:43.245695 22275 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:10:43.245872 22275 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:10:43.250443 22275 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34017
I20260810 04:10:43.250478 22351 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34017 every 8 connection(s)
I20260810 04:10:43.250818 22275 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786334794676589-31872-0/minicluster-data/master-0/data/info.pb
I20260810 04:10:43.251797 22352 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:10:43.254232 22352 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed6d09f42a864c8e92c15ff52d3498a9: Bootstrap starting.
I20260810 04:10:43.254911 22352 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ed6d09f42a864c8e92c15ff52d3498a9: Neither blocks nor log segments found. Creating new log.
I20260810 04:10:43.255297 22352 log.cc:824] T 00000000000000000000000000000000 P ed6d09f42a864c8e92c15ff52d3498a9: Log is configured to *not* fsync() on all Append() calls
I20260810 04:10:43.256322 22352 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed6d09f42a864c8e92c15ff52d3498a9: No bootstrap required, opened a new log
I20260810 04:10:43.257320 31872 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskws89Io/build/release/bin/kudu as pid 22275
I20260810 04:10:43.257459 31872 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786334794676589-31872-0/minicluster-data/master-0/wal/instance
I20260810 04:10:43.257650 31872 external_mini_cluster.cc:1428] Setting key 6696cf7f1414230829c2ff1ba41ce83c
I20260810 04:10:43.258846 22352 raft_consensus.cc:348] T 00000000000000000000000000000000 P ed6d09f42a864c8e92c15ff52d3498a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed6d09f42a864c8e92c15ff52d3498a9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34017 } }
I20260810 04:10:43.258993 22352 raft_consensus.cc:374] T 00000000000000000000000000000000 P ed6d09f42a864c8e92c15ff52d3498a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:10:43.259063 22352 raft_consensus.cc:729] T 00000000000000000000000000000000 P ed6d09f42a864c8e92c15ff52d3498a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed6d09f42a864c8e92c15ff52d3498a9, State: Initialized, Role: FOLLOWER
I20260810 04:10:43.259191 22352 consensus_queue.cc:260] T 00000000000000000000000000000000 P ed6d09f42a864c8e92c15ff52d3498a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed6d09f42a864c8e92c15ff52d3498a9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34017 } }
I20260810 04:10:43.259276 22352 raft_consensus.cc:388] T 00000000000000000000000000000000 P ed6d09f42a864c8e92c15ff52d3498a9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 04:10:43.259310 22352 raft_consensus.cc:482] T 00000000000000000000000000000000 P ed6d09f42a864c8e92c15ff52d3498a9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 04:10:43.259359 22352 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ed6d09f42a864c8e92c15ff52d3498a9 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:10:43.260607 22352 raft_consensus.cc:504] T 00000000000000000000000000000000 P ed6d09f42a864c8e92c15ff52d3498a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed6d09f42a864c8e92c15ff52d3498a9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34017 } }
I20260810 04:10:43.260743 22352 leader_election.cc:302] T 00000000000000000000000000000000 P ed6d09f42a864c8e92c15ff52d3498a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ed6d09f42a864c8e92c15ff52d3498a9; no voters:
I20260810 04:10:43.260922 22352 leader_election.cc:288] T 00000000000000000000000000000000 P ed6d09f42a864c8e92c15ff52d3498a9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 04:10:43.260960 22357 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ed6d09f42a864c8e92c15ff52d3498a9 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:10:43.261073 22357 raft_consensus.cc:686] T 00000000000000000000000000000000 P ed6d09f42a864c8e92c15ff52d3498a9 [term 1 LEADER]: Becoming Leader. State: Replica: ed6d09f42a864c8e92c15ff52d3498a9, State: Running, Role: LEADER
I20260810 04:10:43.261149 22357 consensus_queue.cc:237] T 00000000000000000000000000000000 P ed6d09f42a864c8e92c15ff52d3498a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed6d09f42a864c8e92c15ff52d3498a9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34017 } }
I20260810 04:10:43.261195 22352 sys_catalog.cc:564] T 00000000000000000000000000000000 P ed6d09f42a864c8e92c15ff52d3498a9 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:10:43.261488 22357 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed6d09f42a864c8e92c15ff52d3498a9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ed6d09f42a864c8e92c15ff52d3498a9. Latest consensus state: current_term: 1 leader_uuid: "ed6d09f42a864c8e92c15ff52d3498a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed6d09f42a864c8e92c15ff52d3498a9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34017 } } }
I20260810 04:10:43.261561 22357 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed6d09f42a864c8e92c15ff52d3498a9 [sys.catalog]: This master's current role is: LEADER
I20260810 04:10:43.261523 22358 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed6d09f42a864c8e92c15ff52d3498a9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ed6d09f42a864c8e92c15ff52d3498a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed6d09f42a864c8e92c15ff52d3498a9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34017 } } }
I20260810 04:10:43.261664 22358 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed6d09f42a864c8e92c15ff52d3498a9 [sys.catalog]: This master's current role is: LEADER
I20260810 04:10:43.262018 22364 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 04:10:43.262413 22356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:43.258272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55043 (local address 127.0.0.1:34017)
0810 04:10:43.260310 (+ 2038us) server_negotiation.cc:206] Beginning negotiation
0810 04:10:43.260320 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 04:10:43.260362 (+ 42us) server_negotiation.cc:407] Connection header received
0810 04:10:43.260499 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:10:43.260509 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:10:43.260541 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:10:43.260826 (+ 285us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:10:43.261539 (+ 713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:10:43.261543 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:10:43.261544 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:10:43.261548 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:10:43.261801 (+ 253us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:10:43.261821 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:10:43.261907 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0810 04:10:43.262120 (+ 213us) server_negotiation.cc:299] Negotiation successful
0810 04:10:43.262169 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1968,"thread_start_us":96,"threads_started":1}
I20260810 04:10:43.262861 22364 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 04:10:43.263335 31872 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 04:10:43.264758 22364 catalog_manager.cc:1347] Generated new cluster ID: a4b42f6ed9fc4fc195f8e413cc1c680f
I20260810 04:10:43.265192 22364 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 04:10:43.266290 22356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:43.265214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56089 (local address 127.0.0.1:34017)
0810 04:10:43.265253 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0810 04:10:43.265255 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:10:43.265264 (+ 9us) server_negotiation.cc:407] Connection header received
0810 04:10:43.265287 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:10:43.265288 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:10:43.265293 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:10:43.265351 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:10:43.266055 (+ 704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:10:43.266057 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:10:43.266057 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:10:43.266061 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:10:43.266076 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:10:43.266081 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:10:43.266112 (+ 31us) server_negotiation.cc:1035] Waiting for connection context
0810 04:10:43.266220 (+ 108us) server_negotiation.cc:299] Negotiation successful
0810 04:10:43.266235 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260810 04:10:43.284392 22364 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 04:10:43.284602 22364 catalog_manager.cc:1504] Loading token signing keys...
I20260810 04:10:43.290956 22364 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ed6d09f42a864c8e92c15ff52d3498a9: Generated new TSK 0
I20260810 04:10:43.291177 22364 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 04:10:43.297588 31872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskws89Io/build/release/bin/kudu with pid 22275
2026-08-10T04:10:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1396 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786334794676589-31872-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 04:10:43 dist-test-slave-8w9v krb5kdc[22389](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 04:10:43 dist-test-slave-8w9v krb5kdc[22389](info): set up 2 sockets
Aug 10 04:10:43 dist-test-slave-8w9v krb5kdc[22389](info): commencing operation
krb5kdc: starting...
W20260810 04:10:44.401620 31872 mini_kdc.cc:121] Time spent starting KDC: real 1.079s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786334794676589-31872-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786334794676589-31872-0/krb5kdc/test-user.keytab.
Aug 10 04:10:44 dist-test-slave-8w9v krb5kdc[22389](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335044, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T04:10:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T04:10:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786334794676589-31872-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786334794676589-31872-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 04:10:44.531189 31872 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskws89Io/build/release/bin/kudu
/tmp/dist-test-taskws89Io/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786334794676589-31872-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786334794676589-31872-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786334794676589-31872-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786334794676589-31872-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42241
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786334794676589-31872-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42241
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786334794676589-31872-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786334794676589-31872-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786334794676589-31872-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:10:44.616608 22403 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:10:44.616879 22403 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:10:44.616922 22403 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:10:44.618151 22403 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:10:44.618230 22403 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:10:44.618258 22403 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:10:44.618295 22403 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:10:44.618320 22403 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:10:44.619879 22403 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786334794676589-31872-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786334794676589-31872-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42241
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786334794676589-31872-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42241
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786334794676589-31872-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22403
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786334794676589-31872-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:10:44.620205 22403 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:10:44.620466 22403 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:10:44.625773 22409 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:10:44.625859 22411 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:10:44.625768 22408 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:10:44.625800 22403 server_base.cc:1034] running on GCE node
I20260810 04:10:44.626309 22403 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:10:44.626642 22403 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:10:44.627817 22403 hybrid_clock.cc:648] HybridClock initialized: now 1786335044627814 us; error 42 us; skew 500 ppm
Aug 10 04:10:44 dist-test-slave-8w9v krb5kdc[22389](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335044, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:10:44.634068 22403 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 04:10:44.634552 22403 webserver.cc:460] Webserver started at http://127.0.0.1:40227/ using document root <none> and password file <none>
I20260810 04:10:44.634817 22403 fs_manager.cc:359] Metadata directory not provided
I20260810 04:10:44.634900 22403 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:10:44.635071 22403 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:10:44.636379 22403 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786334794676589-31872-0/minicluster-data/master-0/data/instance:
uuid: "e16548dcda984efcb0f7eecd6a98ecf1"
format_stamp: "Formatted at 2026-08-10 04:10:44 on dist-test-slave-8w9v"
server_key: "893db2e937e33050f5049241367f3aaa"
server_key_iv: "829b9e160fd7299a9c187c307a6b7fa9"
server_key_version: "encryptionkey@0"
I20260810 04:10:44.636785 22403 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786334794676589-31872-0/minicluster-data/master-0/wal/instance:
uuid: "e16548dcda984efcb0f7eecd6a98ecf1"
format_stamp: "Formatted at 2026-08-10 04:10:44 on dist-test-slave-8w9v"
server_key: "893db2e937e33050f5049241367f3aaa"
server_key_iv: "829b9e160fd7299a9c187c307a6b7fa9"
server_key_version: "encryptionkey@0"
I20260810 04:10:44.638278 22403 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260810 04:10:44.639149 22419 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:10:44.639346 22403 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:10:44.639475 22403 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786334794676589-31872-0/minicluster-data/master-0/data,/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786334794676589-31872-0/minicluster-data/master-0/wal
uuid: "e16548dcda984efcb0f7eecd6a98ecf1"
format_stamp: "Formatted at 2026-08-10 04:10:44 on dist-test-slave-8w9v"
server_key: "893db2e937e33050f5049241367f3aaa"
server_key_iv: "829b9e160fd7299a9c187c307a6b7fa9"
server_key_version: "encryptionkey@0"
I20260810 04:10:44.639564 22403 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786334794676589-31872-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786334794676589-31872-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786334794676589-31872-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:10:44.697767 22403 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:10:44.700068 22403 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:10:44.700291 22403 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:10:44.705482 22403 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42241
I20260810 04:10:44.705530 22479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42241 every 8 connection(s)
I20260810 04:10:44.705948 22403 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786334794676589-31872-0/minicluster-data/master-0/data/info.pb
I20260810 04:10:44.706851 22480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:10:44.707533 31872 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskws89Io/build/release/bin/kudu as pid 22403
I20260810 04:10:44.707677 31872 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786334794676589-31872-0/minicluster-data/master-0/wal/instance
I20260810 04:10:44.707901 31872 external_mini_cluster.cc:1428] Setting key a31798c31dc91a7adf2eb86b1c551080
I20260810 04:10:44.710196 22480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e16548dcda984efcb0f7eecd6a98ecf1: Bootstrap starting.
I20260810 04:10:44.710822 22480 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e16548dcda984efcb0f7eecd6a98ecf1: Neither blocks nor log segments found. Creating new log.
I20260810 04:10:44.711144 22480 log.cc:824] T 00000000000000000000000000000000 P e16548dcda984efcb0f7eecd6a98ecf1: Log is configured to *not* fsync() on all Append() calls
I20260810 04:10:44.711836 22480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e16548dcda984efcb0f7eecd6a98ecf1: No bootstrap required, opened a new log
I20260810 04:10:44.713723 22480 raft_consensus.cc:348] T 00000000000000000000000000000000 P e16548dcda984efcb0f7eecd6a98ecf1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e16548dcda984efcb0f7eecd6a98ecf1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42241 } }
I20260810 04:10:44.713917 22480 raft_consensus.cc:374] T 00000000000000000000000000000000 P e16548dcda984efcb0f7eecd6a98ecf1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:10:44.713992 22480 raft_consensus.cc:729] T 00000000000000000000000000000000 P e16548dcda984efcb0f7eecd6a98ecf1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e16548dcda984efcb0f7eecd6a98ecf1, State: Initialized, Role: FOLLOWER
I20260810 04:10:44.714139 22480 consensus_queue.cc:260] T 00000000000000000000000000000000 P e16548dcda984efcb0f7eecd6a98ecf1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e16548dcda984efcb0f7eecd6a98ecf1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42241 } }
I20260810 04:10:44.714246 22480 raft_consensus.cc:388] T 00000000000000000000000000000000 P e16548dcda984efcb0f7eecd6a98ecf1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 04:10:44.714290 22480 raft_consensus.cc:482] T 00000000000000000000000000000000 P e16548dcda984efcb0f7eecd6a98ecf1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 04:10:44.714341 22480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e16548dcda984efcb0f7eecd6a98ecf1 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:10:44.714936 22480 raft_consensus.cc:504] T 00000000000000000000000000000000 P e16548dcda984efcb0f7eecd6a98ecf1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e16548dcda984efcb0f7eecd6a98ecf1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42241 } }
I20260810 04:10:44.715075 22480 leader_election.cc:302] T 00000000000000000000000000000000 P e16548dcda984efcb0f7eecd6a98ecf1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e16548dcda984efcb0f7eecd6a98ecf1; no voters:
I20260810 04:10:44.715229 22480 leader_election.cc:288] T 00000000000000000000000000000000 P e16548dcda984efcb0f7eecd6a98ecf1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 04:10:44.715297 22486 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e16548dcda984efcb0f7eecd6a98ecf1 [term 1 FOLLOWER]: Leader election won for term 1
Aug 10 04:10:44 dist-test-slave-8w9v krb5kdc[22389](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335044, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260810 04:10:44.715492 22480 sys_catalog.cc:564] T 00000000000000000000000000000000 P e16548dcda984efcb0f7eecd6a98ecf1 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:10:44.715498 22486 raft_consensus.cc:686] T 00000000000000000000000000000000 P e16548dcda984efcb0f7eecd6a98ecf1 [term 1 LEADER]: Becoming Leader. State: Replica: e16548dcda984efcb0f7eecd6a98ecf1, State: Running, Role: LEADER
I20260810 04:10:44.715641 22486 consensus_queue.cc:237] T 00000000000000000000000000000000 P e16548dcda984efcb0f7eecd6a98ecf1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e16548dcda984efcb0f7eecd6a98ecf1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42241 } }
I20260810 04:10:44.716092 22487 sys_catalog.cc:455] T 00000000000000000000000000000000 P e16548dcda984efcb0f7eecd6a98ecf1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e16548dcda984efcb0f7eecd6a98ecf1. Latest consensus state: current_term: 1 leader_uuid: "e16548dcda984efcb0f7eecd6a98ecf1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e16548dcda984efcb0f7eecd6a98ecf1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42241 } } }
I20260810 04:10:44.716259 22488 sys_catalog.cc:455] T 00000000000000000000000000000000 P e16548dcda984efcb0f7eecd6a98ecf1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e16548dcda984efcb0f7eecd6a98ecf1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e16548dcda984efcb0f7eecd6a98ecf1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42241 } } }
I20260810 04:10:44.716289 22487 sys_catalog.cc:458] T 00000000000000000000000000000000 P e16548dcda984efcb0f7eecd6a98ecf1 [sys.catalog]: This master's current role is: LEADER
I20260810 04:10:44.716434 22488 sys_catalog.cc:458] T 00000000000000000000000000000000 P e16548dcda984efcb0f7eecd6a98ecf1 [sys.catalog]: This master's current role is: LEADER
I20260810 04:10:44.716673 22496 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 04:10:44.717471 22496 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 04:10:44.719172 22496 catalog_manager.cc:1347] Generated new cluster ID: 360215ad4cdc4e728668e434c90de450
I20260810 04:10:44.719225 22496 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 04:10:44.720045 22483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:44.708554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.214:43051 (local address 127.0.0.1:42241)
0810 04:10:44.708761 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0810 04:10:44.708771 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 04:10:44.708808 (+ 37us) server_negotiation.cc:407] Connection header received
0810 04:10:44.709398 (+ 590us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:10:44.709408 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:10:44.709496 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:10:44.709666 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:10:44.710457 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:44.711557 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:44.712512 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:44.712974 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:44.713114 (+ 140us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:10:44.716180 (+ 3066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:10:44.716184 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:10:44.716185 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:10:44.716197 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:10:44.718228 (+ 2031us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:10:44.718743 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:10:44.718747 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:10:44.718747 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:10:44.718867 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:10:44.719152 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:10:44.719153 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:10:44.719154 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:10:44.719457 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:10:44.719646 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0810 04:10:44.719765 (+ 119us) server_negotiation.cc:299] Negotiation successful
0810 04:10:44.719870 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":150,"thread_start_us":94,"threads_started":1}
I20260810 04:10:44.721127 31872 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 04:10:44.734977 22483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:44.728456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.214:49679 (local address 127.0.0.1:42241)
0810 04:10:44.728509 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0810 04:10:44.728511 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:10:44.728633 (+ 122us) server_negotiation.cc:407] Connection header received
0810 04:10:44.728707 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:10:44.728708 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:10:44.728718 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:10:44.728773 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:10:44.729340 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:44.729825 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:44.730548 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:44.730787 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:44.730881 (+ 94us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:10:44.732191 (+ 1310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:10:44.732192 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:10:44.732193 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:10:44.732201 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:10:44.733736 (+ 1535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:10:44.734129 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:10:44.734130 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:10:44.734130 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:10:44.734190 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:10:44.734434 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:10:44.734435 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:10:44.734436 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:10:44.734555 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:10:44.734687 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0810 04:10:44.734800 (+ 113us) server_negotiation.cc:299] Negotiation successful
0810 04:10:44.734863 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":31}
I20260810 04:10:44.747100 22496 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 04:10:44.747818 22496 catalog_manager.cc:1504] Loading token signing keys...
I20260810 04:10:44.754698 22496 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e16548dcda984efcb0f7eecd6a98ecf1: Generated new TSK 0
I20260810 04:10:44.754886 22496 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 04:10:44.757302 31872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskws89Io/build/release/bin/kudu with pid 22403
W20260810 04:10:44.764937 22386 connection.cc:535] client connection to 127.0.0.1:42241 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42241): Connection reset by peer (error 104)
2026-08-10T04:10:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1456 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786334794676589-31872-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 04:10:44 dist-test-slave-8w9v krb5kdc[22516](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 04:10:44 dist-test-slave-8w9v krb5kdc[22516](info): set up 2 sockets
Aug 10 04:10:44 dist-test-slave-8w9v krb5kdc[22516](info): commencing operation
krb5kdc: starting...
W20260810 04:10:45.851405 31872 mini_kdc.cc:121] Time spent starting KDC: real 1.073s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786334794676589-31872-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786334794676589-31872-0/krb5kdc/test-user.keytab.
Aug 10 04:10:45 dist-test-slave-8w9v krb5kdc[22516](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335045, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T04:10:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T04:10:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786334794676589-31872-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786334794676589-31872-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 04:10:45.979544 31872 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskws89Io/build/release/bin/kudu
/tmp/dist-test-taskws89Io/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786334794676589-31872-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786334794676589-31872-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786334794676589-31872-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786334794676589-31872-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44325
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786334794676589-31872-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44325
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786334794676589-31872-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786334794676589-31872-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786334794676589-31872-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:10:46.067695 22533 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:10:46.067960 22533 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:10:46.068004 22533 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:10:46.069284 22533 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:10:46.069365 22533 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:10:46.069392 22533 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:10:46.069415 22533 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:10:46.069445 22533 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:10:46.071039 22533 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786334794676589-31872-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786334794676589-31872-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44325
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786334794676589-31872-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44325
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786334794676589-31872-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22533
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786334794676589-31872-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:10:46.071353 22533 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:10:46.071607 22533 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:10:46.076930 22539 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:10:46.076939 22541 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:10:46.076939 22538 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:10:46.077307 22533 server_base.cc:1034] running on GCE node
I20260810 04:10:46.077605 22533 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:10:46.077917 22533 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:10:46.079083 22533 hybrid_clock.cc:648] HybridClock initialized: now 1786335046079070 us; error 47 us; skew 500 ppm
Aug 10 04:10:46 dist-test-slave-8w9v krb5kdc[22516](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335046, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:10:46.081763 22533 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 04:10:46.082213 22533 webserver.cc:460] Webserver started at http://127.0.0.1:33937/ using document root <none> and password file <none>
I20260810 04:10:46.082463 22533 fs_manager.cc:359] Metadata directory not provided
I20260810 04:10:46.082533 22533 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:10:46.082680 22533 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:10:46.083734 22533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786334794676589-31872-0/minicluster-data/master-0/data/instance:
uuid: "f6bff76a140d4f00a16c1bd6dff45f43"
format_stamp: "Formatted at 2026-08-10 04:10:46 on dist-test-slave-8w9v"
server_key: "8c814d83431c80703ed5429fd6df386c"
server_key_iv: "689b914801f75c72889cb61cefe328d6"
server_key_version: "encryptionkey@0"
I20260810 04:10:46.084105 22533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786334794676589-31872-0/minicluster-data/master-0/wal/instance:
uuid: "f6bff76a140d4f00a16c1bd6dff45f43"
format_stamp: "Formatted at 2026-08-10 04:10:46 on dist-test-slave-8w9v"
server_key: "8c814d83431c80703ed5429fd6df386c"
server_key_iv: "689b914801f75c72889cb61cefe328d6"
server_key_version: "encryptionkey@0"
I20260810 04:10:46.085543 22533 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:10:46.086278 22548 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:10:46.086458 22533 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 04:10:46.086552 22533 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786334794676589-31872-0/minicluster-data/master-0/data,/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786334794676589-31872-0/minicluster-data/master-0/wal
uuid: "f6bff76a140d4f00a16c1bd6dff45f43"
format_stamp: "Formatted at 2026-08-10 04:10:46 on dist-test-slave-8w9v"
server_key: "8c814d83431c80703ed5429fd6df386c"
server_key_iv: "689b914801f75c72889cb61cefe328d6"
server_key_version: "encryptionkey@0"
I20260810 04:10:46.086632 22533 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786334794676589-31872-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786334794676589-31872-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786334794676589-31872-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:10:46.108443 22533 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:10:46.110996 22533 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:10:46.111171 22533 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:10:46.115477 22533 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44325
I20260810 04:10:46.115517 22606 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44325 every 8 connection(s)
I20260810 04:10:46.115911 22533 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786334794676589-31872-0/minicluster-data/master-0/data/info.pb
I20260810 04:10:46.116940 22607 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:10:46.119267 22607 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6bff76a140d4f00a16c1bd6dff45f43: Bootstrap starting.
I20260810 04:10:46.119866 22607 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f6bff76a140d4f00a16c1bd6dff45f43: Neither blocks nor log segments found. Creating new log.
I20260810 04:10:46.120198 22607 log.cc:824] T 00000000000000000000000000000000 P f6bff76a140d4f00a16c1bd6dff45f43: Log is configured to *not* fsync() on all Append() calls
I20260810 04:10:46.120954 22607 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6bff76a140d4f00a16c1bd6dff45f43: No bootstrap required, opened a new log
I20260810 04:10:46.122579 22607 raft_consensus.cc:348] T 00000000000000000000000000000000 P f6bff76a140d4f00a16c1bd6dff45f43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6bff76a140d4f00a16c1bd6dff45f43" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44325 } }
I20260810 04:10:46.122717 22607 raft_consensus.cc:374] T 00000000000000000000000000000000 P f6bff76a140d4f00a16c1bd6dff45f43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:10:46.122740 22607 raft_consensus.cc:729] T 00000000000000000000000000000000 P f6bff76a140d4f00a16c1bd6dff45f43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6bff76a140d4f00a16c1bd6dff45f43, State: Initialized, Role: FOLLOWER
I20260810 04:10:46.122941 22607 consensus_queue.cc:260] T 00000000000000000000000000000000 P f6bff76a140d4f00a16c1bd6dff45f43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6bff76a140d4f00a16c1bd6dff45f43" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44325 } }
I20260810 04:10:46.123008 22607 raft_consensus.cc:388] T 00000000000000000000000000000000 P f6bff76a140d4f00a16c1bd6dff45f43 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 04:10:46.123030 22607 raft_consensus.cc:482] T 00000000000000000000000000000000 P f6bff76a140d4f00a16c1bd6dff45f43 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 04:10:46.123098 22607 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f6bff76a140d4f00a16c1bd6dff45f43 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:10:46.123646 22607 raft_consensus.cc:504] T 00000000000000000000000000000000 P f6bff76a140d4f00a16c1bd6dff45f43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6bff76a140d4f00a16c1bd6dff45f43" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44325 } }
I20260810 04:10:46.123760 22607 leader_election.cc:302] T 00000000000000000000000000000000 P f6bff76a140d4f00a16c1bd6dff45f43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f6bff76a140d4f00a16c1bd6dff45f43; no voters:
I20260810 04:10:46.123956 22607 leader_election.cc:288] T 00000000000000000000000000000000 P f6bff76a140d4f00a16c1bd6dff45f43 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 04:10:46.123994 22610 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f6bff76a140d4f00a16c1bd6dff45f43 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:10:46.124174 22610 raft_consensus.cc:686] T 00000000000000000000000000000000 P f6bff76a140d4f00a16c1bd6dff45f43 [term 1 LEADER]: Becoming Leader. State: Replica: f6bff76a140d4f00a16c1bd6dff45f43, State: Running, Role: LEADER
I20260810 04:10:46.124239 22607 sys_catalog.cc:564] T 00000000000000000000000000000000 P f6bff76a140d4f00a16c1bd6dff45f43 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:10:46.124344 22610 consensus_queue.cc:237] T 00000000000000000000000000000000 P f6bff76a140d4f00a16c1bd6dff45f43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6bff76a140d4f00a16c1bd6dff45f43" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44325 } }
I20260810 04:10:46.124894 22611 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6bff76a140d4f00a16c1bd6dff45f43 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f6bff76a140d4f00a16c1bd6dff45f43" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6bff76a140d4f00a16c1bd6dff45f43" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44325 } } }
I20260810 04:10:46.124972 22611 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6bff76a140d4f00a16c1bd6dff45f43 [sys.catalog]: This master's current role is: LEADER
I20260810 04:10:46.125267 31872 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskws89Io/build/release/bin/kudu as pid 22533
I20260810 04:10:46.125419 31872 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786334794676589-31872-0/minicluster-data/master-0/wal/instance
I20260810 04:10:46.125367 22612 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6bff76a140d4f00a16c1bd6dff45f43 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f6bff76a140d4f00a16c1bd6dff45f43. Latest consensus state: current_term: 1 leader_uuid: "f6bff76a140d4f00a16c1bd6dff45f43" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6bff76a140d4f00a16c1bd6dff45f43" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44325 } } }
I20260810 04:10:46.125451 22612 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6bff76a140d4f00a16c1bd6dff45f43 [sys.catalog]: This master's current role is: LEADER
I20260810 04:10:46.125654 31872 external_mini_cluster.cc:1428] Setting key a6ab67a96936aa5a14ff68b5fcf51246
W20260810 04:10:46.126366 22627 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f6bff76a140d4f00a16c1bd6dff45f43: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 04:10:46.126420 22627 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 04:10:46.126474 22619 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 04:10:46.126647 22619 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 04:10:46.128201 22619 catalog_manager.cc:1347] Generated new cluster ID: fd21cf06b1f840a297a39c3e3308746e
I20260810 04:10:46.128255 22619 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 04:10:46.131135 22631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:46.126198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.214:55007 (local address 127.0.0.1:44325)
0810 04:10:46.126405 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0810 04:10:46.126415 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 04:10:46.126643 (+ 228us) server_negotiation.cc:407] Connection header received
0810 04:10:46.126821 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:10:46.126827 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:10:46.126873 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:10:46.127035 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:10:46.127206 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:46.128374 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:46.129164 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:46.129612 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:46.129748 (+ 136us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:10:46.130494 (+ 746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:10:46.130498 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:10:46.130498 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:10:46.130676 (+ 178us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:10:46.130706 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:10:46.130716 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:10:46.130815 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0810 04:10:46.130905 (+ 90us) server_negotiation.cc:299] Negotiation successful
0810 04:10:46.130934 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":93,"threads_started":1}
I20260810 04:10:46.132084 31872 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 04:10:46.143558 22631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:46.133432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.214:55623 (local address 127.0.0.1:44325)
0810 04:10:46.133467 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0810 04:10:46.133468 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 04:10:46.140356 (+ 6888us) server_negotiation.cc:407] Connection header received
0810 04:10:46.140445 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:10:46.140446 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:10:46.140457 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:10:46.140536 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:10:46.140714 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:46.141310 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:46.142112 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:46.142353 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:46.142441 (+ 88us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:10:46.143213 (+ 772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:10:46.143214 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:10:46.143215 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:10:46.143220 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:10:46.143246 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:10:46.143250 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:10:46.143344 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0810 04:10:46.143415 (+ 71us) server_negotiation.cc:299] Negotiation successful
0810 04:10:46.143429 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":17}
I20260810 04:10:46.148691 22619 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 04:10:46.149407 22619 catalog_manager.cc:1504] Loading token signing keys...
I20260810 04:10:46.155507 22619 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f6bff76a140d4f00a16c1bd6dff45f43: Generated new TSK 0
I20260810 04:10:46.155686 22619 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 04:10:46.162648 31872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskws89Io/build/release/bin/kudu with pid 22533
W20260810 04:10:46.169709 22514 connection.cc:535] client connection to 127.0.0.1:44325 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44325): Connection reset by peer (error 104)
2026-08-10T04:10:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1404 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786334794676589-31872-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 04:10:46 dist-test-slave-8w9v krb5kdc[22640](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 04:10:46 dist-test-slave-8w9v krb5kdc[22640](info): set up 2 sockets
Aug 10 04:10:46 dist-test-slave-8w9v krb5kdc[22640](info): commencing operation
krb5kdc: starting...
W20260810 04:10:47.268505 31872 mini_kdc.cc:121] Time spent starting KDC: real 1.085s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786334794676589-31872-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786334794676589-31872-0/krb5kdc/test-user.keytab.
Aug 10 04:10:47 dist-test-slave-8w9v krb5kdc[22640](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335047, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T04:10:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T04:10:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786334794676589-31872-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786334794676589-31872-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 04:10:47.391484 31872 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskws89Io/build/release/bin/kudu
/tmp/dist-test-taskws89Io/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786334794676589-31872-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786334794676589-31872-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786334794676589-31872-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786334794676589-31872-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44643
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786334794676589-31872-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44643
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786334794676589-31872-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786334794676589-31872-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786334794676589-31872-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:10:47.475426 22655 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:10:47.475622 22655 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:10:47.475644 22655 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:10:47.476805 22655 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:10:47.476848 22655 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:10:47.476862 22655 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:10:47.476876 22655 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:10:47.476888 22655 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:10:47.478287 22655 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786334794676589-31872-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786334794676589-31872-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44643
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786334794676589-31872-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44643
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786334794676589-31872-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22655
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786334794676589-31872-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:10:47.478472 22655 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:10:47.478637 22655 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:10:47.483624 22663 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:10:47.483672 22655 server_base.cc:1034] running on GCE node
W20260810 04:10:47.483628 22660 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:10:47.483645 22661 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:10:47.484113 22655 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:10:47.484459 22655 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:10:47.485607 22655 hybrid_clock.cc:648] HybridClock initialized: now 1786335047485596 us; error 27 us; skew 500 ppm
Aug 10 04:10:47 dist-test-slave-8w9v krb5kdc[22640](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335047, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:10:47.488035 22655 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 04:10:47.488479 22655 webserver.cc:460] Webserver started at http://127.0.0.1:45995/ using document root <none> and password file <none>
I20260810 04:10:47.488689 22655 fs_manager.cc:359] Metadata directory not provided
I20260810 04:10:47.488730 22655 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:10:47.488816 22655 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:10:47.489810 22655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786334794676589-31872-0/minicluster-data/master-0/data/instance:
uuid: "25f530a8e88a4b7cbfe726e73577cbae"
format_stamp: "Formatted at 2026-08-10 04:10:47 on dist-test-slave-8w9v"
server_key: "221c1a299313589d88c4644b999eb64c"
server_key_iv: "b21b6e34aa5011635a45091797bfa4c9"
server_key_version: "encryptionkey@0"
I20260810 04:10:47.490115 22655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786334794676589-31872-0/minicluster-data/master-0/wal/instance:
uuid: "25f530a8e88a4b7cbfe726e73577cbae"
format_stamp: "Formatted at 2026-08-10 04:10:47 on dist-test-slave-8w9v"
server_key: "221c1a299313589d88c4644b999eb64c"
server_key_iv: "b21b6e34aa5011635a45091797bfa4c9"
server_key_version: "encryptionkey@0"
I20260810 04:10:47.491380 22655 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 04:10:47.492053 22670 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:10:47.492254 22655 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260810 04:10:47.492359 22655 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786334794676589-31872-0/minicluster-data/master-0/data,/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786334794676589-31872-0/minicluster-data/master-0/wal
uuid: "25f530a8e88a4b7cbfe726e73577cbae"
format_stamp: "Formatted at 2026-08-10 04:10:47 on dist-test-slave-8w9v"
server_key: "221c1a299313589d88c4644b999eb64c"
server_key_iv: "b21b6e34aa5011635a45091797bfa4c9"
server_key_version: "encryptionkey@0"
I20260810 04:10:47.492416 22655 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786334794676589-31872-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786334794676589-31872-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786334794676589-31872-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:10:47.495638 22655 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:10:47.497618 22655 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:10:47.497723 22655 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:10:47.501618 22655 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44643
I20260810 04:10:47.501667 22736 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44643 every 8 connection(s)
I20260810 04:10:47.501994 22655 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786334794676589-31872-0/minicluster-data/master-0/data/info.pb
I20260810 04:10:47.502909 22737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:10:47.505205 22737 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25f530a8e88a4b7cbfe726e73577cbae: Bootstrap starting.
I20260810 04:10:47.505777 22737 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 25f530a8e88a4b7cbfe726e73577cbae: Neither blocks nor log segments found. Creating new log.
I20260810 04:10:47.506079 22737 log.cc:824] T 00000000000000000000000000000000 P 25f530a8e88a4b7cbfe726e73577cbae: Log is configured to *not* fsync() on all Append() calls
I20260810 04:10:47.506353 31872 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskws89Io/build/release/bin/kudu as pid 22655
I20260810 04:10:47.506477 31872 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786334794676589-31872-0/minicluster-data/master-0/wal/instance
I20260810 04:10:47.506686 31872 external_mini_cluster.cc:1428] Setting key 08363003b93972b7a2ee4e61b3b49c66
I20260810 04:10:47.506908 22737 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25f530a8e88a4b7cbfe726e73577cbae: No bootstrap required, opened a new log
I20260810 04:10:47.509305 22737 raft_consensus.cc:348] T 00000000000000000000000000000000 P 25f530a8e88a4b7cbfe726e73577cbae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25f530a8e88a4b7cbfe726e73577cbae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44643 } }
I20260810 04:10:47.509464 22737 raft_consensus.cc:374] T 00000000000000000000000000000000 P 25f530a8e88a4b7cbfe726e73577cbae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:10:47.509512 22737 raft_consensus.cc:729] T 00000000000000000000000000000000 P 25f530a8e88a4b7cbfe726e73577cbae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25f530a8e88a4b7cbfe726e73577cbae, State: Initialized, Role: FOLLOWER
I20260810 04:10:47.509668 22744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:47.507195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.214:60557 (local address 127.0.0.1:44643)
0810 04:10:47.507389 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0810 04:10:47.507396 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 04:10:47.507420 (+ 24us) server_negotiation.cc:407] Connection header received
0810 04:10:47.508228 (+ 808us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:10:47.508235 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:10:47.508262 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:10:47.508395 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:10:47.508995 (+ 600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:10:47.508998 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:10:47.508998 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:10:47.509002 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:10:47.509038 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:10:47.509048 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:10:47.509105 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0810 04:10:47.509186 (+ 81us) server_negotiation.cc:299] Negotiation successful
0810 04:10:47.509241 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":66,"threads_started":1}
I20260810 04:10:47.509634 22737 consensus_queue.cc:260] T 00000000000000000000000000000000 P 25f530a8e88a4b7cbfe726e73577cbae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25f530a8e88a4b7cbfe726e73577cbae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44643 } }
I20260810 04:10:47.509735 22737 raft_consensus.cc:388] T 00000000000000000000000000000000 P 25f530a8e88a4b7cbfe726e73577cbae [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 04:10:47.509778 22737 raft_consensus.cc:482] T 00000000000000000000000000000000 P 25f530a8e88a4b7cbfe726e73577cbae [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 04:10:47.509833 22737 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 25f530a8e88a4b7cbfe726e73577cbae [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:10:47.510504 22737 raft_consensus.cc:504] T 00000000000000000000000000000000 P 25f530a8e88a4b7cbfe726e73577cbae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25f530a8e88a4b7cbfe726e73577cbae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44643 } }
I20260810 04:10:47.510622 22737 leader_election.cc:302] T 00000000000000000000000000000000 P 25f530a8e88a4b7cbfe726e73577cbae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 25f530a8e88a4b7cbfe726e73577cbae; no voters:
I20260810 04:10:47.510649 31872 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 04:10:47.510794 22737 leader_election.cc:288] T 00000000000000000000000000000000 P 25f530a8e88a4b7cbfe726e73577cbae [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 04:10:47.510819 22746 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 25f530a8e88a4b7cbfe726e73577cbae [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:10:47.510944 22746 raft_consensus.cc:686] T 00000000000000000000000000000000 P 25f530a8e88a4b7cbfe726e73577cbae [term 1 LEADER]: Becoming Leader. State: Replica: 25f530a8e88a4b7cbfe726e73577cbae, State: Running, Role: LEADER
I20260810 04:10:47.511036 22737 sys_catalog.cc:564] T 00000000000000000000000000000000 P 25f530a8e88a4b7cbfe726e73577cbae [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:10:47.511024 22746 consensus_queue.cc:237] T 00000000000000000000000000000000 P 25f530a8e88a4b7cbfe726e73577cbae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25f530a8e88a4b7cbfe726e73577cbae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44643 } }
I20260810 04:10:47.511317 22746 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25f530a8e88a4b7cbfe726e73577cbae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "25f530a8e88a4b7cbfe726e73577cbae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25f530a8e88a4b7cbfe726e73577cbae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44643 } } }
I20260810 04:10:47.511391 22746 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25f530a8e88a4b7cbfe726e73577cbae [sys.catalog]: This master's current role is: LEADER
I20260810 04:10:47.511516 22746 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25f530a8e88a4b7cbfe726e73577cbae [sys.catalog]: SysCatalogTable state changed. Reason: New leader 25f530a8e88a4b7cbfe726e73577cbae. Latest consensus state: current_term: 1 leader_uuid: "25f530a8e88a4b7cbfe726e73577cbae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25f530a8e88a4b7cbfe726e73577cbae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44643 } } }
I20260810 04:10:47.511584 22746 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25f530a8e88a4b7cbfe726e73577cbae [sys.catalog]: This master's current role is: LEADER
W20260810 04:10:47.513039 22770 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 25f530a8e88a4b7cbfe726e73577cbae: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 04:10:47.513096 22770 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 04:10:47.513428 22752 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 04:10:47.513512 22744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:47.512309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.214:34235 (local address 127.0.0.1:44643)
0810 04:10:47.512344 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0810 04:10:47.512345 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 04:10:47.512354 (+ 9us) server_negotiation.cc:407] Connection header received
0810 04:10:47.512397 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:10:47.512398 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:10:47.512402 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:10:47.512451 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:10:47.513222 (+ 771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:10:47.513223 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:10:47.513223 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:10:47.513228 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:10:47.513243 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:10:47.513247 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:10:47.513279 (+ 32us) server_negotiation.cc:1035] Waiting for connection context
0810 04:10:47.513398 (+ 119us) server_negotiation.cc:299] Negotiation successful
0810 04:10:47.513413 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260810 04:10:47.513623 22752 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 04:10:47.515092 22752 catalog_manager.cc:1347] Generated new cluster ID: 8786fb684262409b8eaa67ceb8336598
I20260810 04:10:47.515144 22752 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 04:10:47.524587 22752 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 04:10:47.524753 22752 catalog_manager.cc:1504] Loading token signing keys...
I20260810 04:10:47.529873 22752 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 25f530a8e88a4b7cbfe726e73577cbae: Generated new TSK 0
I20260810 04:10:47.530052 22752 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 04:10:47.531520 31872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskws89Io/build/release/bin/kudu with pid 22655
2026-08-10T04:10:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1369 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786334794676589-31872-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 04:10:47 dist-test-slave-8w9v krb5kdc[22776](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 04:10:47 dist-test-slave-8w9v krb5kdc[22776](info): set up 2 sockets
Aug 10 04:10:47 dist-test-slave-8w9v krb5kdc[22776](info): commencing operation
krb5kdc: starting...
W20260810 04:10:48.621102 31872 mini_kdc.cc:121] Time spent starting KDC: real 1.068s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786334794676589-31872-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786334794676589-31872-0/krb5kdc/test-user.keytab.
Aug 10 04:10:48 dist-test-slave-8w9v krb5kdc[22776](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335048, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T04:10:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T04:10:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786334794676589-31872-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786334794676589-31872-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 04:10:48.742590 31872 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskws89Io/build/release/bin/kudu
/tmp/dist-test-taskws89Io/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786334794676589-31872-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786334794676589-31872-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786334794676589-31872-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786334794676589-31872-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35913
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786334794676589-31872-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35913
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786334794676589-31872-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786334794676589-31872-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786334794676589-31872-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:10:48.824046 22789 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:10:48.824260 22789 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:10:48.824282 22789 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:10:48.825438 22789 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:10:48.825480 22789 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:10:48.825495 22789 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:10:48.825508 22789 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:10:48.825520 22789 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:10:48.827008 22789 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786334794676589-31872-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786334794676589-31872-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35913
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786334794676589-31872-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35913
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786334794676589-31872-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22789
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786334794676589-31872-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:10:48.827250 22789 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:10:48.827473 22789 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:10:48.832680 22796 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:10:48.832653 22798 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:10:48.832670 22795 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:10:48.832851 22789 server_base.cc:1034] running on GCE node
I20260810 04:10:48.833170 22789 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:10:48.833509 22789 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:10:48.834662 22789 hybrid_clock.cc:648] HybridClock initialized: now 1786335048834647 us; error 37 us; skew 500 ppm
Aug 10 04:10:48 dist-test-slave-8w9v krb5kdc[22776](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335048, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:10:48.837253 22789 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 04:10:48.837708 22789 webserver.cc:460] Webserver started at http://127.0.0.1:41795/ using document root <none> and password file <none>
I20260810 04:10:48.837962 22789 fs_manager.cc:359] Metadata directory not provided
I20260810 04:10:48.838016 22789 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:10:48.838181 22789 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:10:48.839210 22789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786334794676589-31872-0/minicluster-data/master-0/data/instance:
uuid: "d0021c3af757495a903b5fdef0d4085b"
format_stamp: "Formatted at 2026-08-10 04:10:48 on dist-test-slave-8w9v"
server_key: "14614ee4a407801430e35789f1187c61"
server_key_iv: "42a24942c271625707e4fe9064de2a0f"
server_key_version: "encryptionkey@0"
I20260810 04:10:48.839571 22789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786334794676589-31872-0/minicluster-data/master-0/wal/instance:
uuid: "d0021c3af757495a903b5fdef0d4085b"
format_stamp: "Formatted at 2026-08-10 04:10:48 on dist-test-slave-8w9v"
server_key: "14614ee4a407801430e35789f1187c61"
server_key_iv: "42a24942c271625707e4fe9064de2a0f"
server_key_version: "encryptionkey@0"
I20260810 04:10:48.841007 22789 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 04:10:48.841761 22805 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:10:48.841938 22789 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 04:10:48.842046 22789 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786334794676589-31872-0/minicluster-data/master-0/data,/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786334794676589-31872-0/minicluster-data/master-0/wal
uuid: "d0021c3af757495a903b5fdef0d4085b"
format_stamp: "Formatted at 2026-08-10 04:10:48 on dist-test-slave-8w9v"
server_key: "14614ee4a407801430e35789f1187c61"
server_key_iv: "42a24942c271625707e4fe9064de2a0f"
server_key_version: "encryptionkey@0"
I20260810 04:10:48.842123 22789 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786334794676589-31872-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786334794676589-31872-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786334794676589-31872-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:10:48.854220 22789 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:10:48.856012 22789 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:10:48.856204 22789 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:10:48.860389 22789 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35913
I20260810 04:10:48.860450 22860 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35913 every 8 connection(s)
I20260810 04:10:48.860802 22789 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786334794676589-31872-0/minicluster-data/master-0/data/info.pb
I20260810 04:10:48.861666 22861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:10:48.863894 22861 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0021c3af757495a903b5fdef0d4085b: Bootstrap starting.
I20260810 04:10:48.864533 22861 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d0021c3af757495a903b5fdef0d4085b: Neither blocks nor log segments found. Creating new log.
I20260810 04:10:48.864830 22861 log.cc:824] T 00000000000000000000000000000000 P d0021c3af757495a903b5fdef0d4085b: Log is configured to *not* fsync() on all Append() calls
I20260810 04:10:48.865530 22861 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0021c3af757495a903b5fdef0d4085b: No bootstrap required, opened a new log
I20260810 04:10:48.867076 22861 raft_consensus.cc:348] T 00000000000000000000000000000000 P d0021c3af757495a903b5fdef0d4085b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0021c3af757495a903b5fdef0d4085b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35913 } }
I20260810 04:10:48.867208 22861 raft_consensus.cc:374] T 00000000000000000000000000000000 P d0021c3af757495a903b5fdef0d4085b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:10:48.867230 22861 raft_consensus.cc:729] T 00000000000000000000000000000000 P d0021c3af757495a903b5fdef0d4085b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0021c3af757495a903b5fdef0d4085b, State: Initialized, Role: FOLLOWER
I20260810 04:10:48.867409 22861 consensus_queue.cc:260] T 00000000000000000000000000000000 P d0021c3af757495a903b5fdef0d4085b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0021c3af757495a903b5fdef0d4085b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35913 } }
I20260810 04:10:48.867473 22861 raft_consensus.cc:388] T 00000000000000000000000000000000 P d0021c3af757495a903b5fdef0d4085b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 04:10:48.867522 22861 raft_consensus.cc:482] T 00000000000000000000000000000000 P d0021c3af757495a903b5fdef0d4085b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 04:10:48.867506 31872 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskws89Io/build/release/bin/kudu as pid 22789
I20260810 04:10:48.867580 22861 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d0021c3af757495a903b5fdef0d4085b [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:10:48.867642 31872 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786334794676589-31872-0/minicluster-data/master-0/wal/instance
I20260810 04:10:48.867826 31872 external_mini_cluster.cc:1428] Setting key 3e4b64ce8e2daa3e1ac97da3db32564b
I20260810 04:10:48.868228 22861 raft_consensus.cc:504] T 00000000000000000000000000000000 P d0021c3af757495a903b5fdef0d4085b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0021c3af757495a903b5fdef0d4085b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35913 } }
I20260810 04:10:48.868362 22861 leader_election.cc:302] T 00000000000000000000000000000000 P d0021c3af757495a903b5fdef0d4085b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d0021c3af757495a903b5fdef0d4085b; no voters:
I20260810 04:10:48.868542 22861 leader_election.cc:288] T 00000000000000000000000000000000 P d0021c3af757495a903b5fdef0d4085b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 04:10:48.868570 22866 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d0021c3af757495a903b5fdef0d4085b [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:10:48.868741 22866 raft_consensus.cc:686] T 00000000000000000000000000000000 P d0021c3af757495a903b5fdef0d4085b [term 1 LEADER]: Becoming Leader. State: Replica: d0021c3af757495a903b5fdef0d4085b, State: Running, Role: LEADER
I20260810 04:10:48.868894 22866 consensus_queue.cc:237] T 00000000000000000000000000000000 P d0021c3af757495a903b5fdef0d4085b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0021c3af757495a903b5fdef0d4085b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35913 } }
I20260810 04:10:48.868862 22861 sys_catalog.cc:564] T 00000000000000000000000000000000 P d0021c3af757495a903b5fdef0d4085b [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:10:48.870213 22869 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0021c3af757495a903b5fdef0d4085b [sys.catalog]: SysCatalogTable state changed. Reason: New leader d0021c3af757495a903b5fdef0d4085b. Latest consensus state: current_term: 1 leader_uuid: "d0021c3af757495a903b5fdef0d4085b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0021c3af757495a903b5fdef0d4085b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35913 } } }
I20260810 04:10:48.870302 22868 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0021c3af757495a903b5fdef0d4085b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d0021c3af757495a903b5fdef0d4085b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0021c3af757495a903b5fdef0d4085b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35913 } } }
I20260810 04:10:48.870401 22869 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0021c3af757495a903b5fdef0d4085b [sys.catalog]: This master's current role is: LEADER
I20260810 04:10:48.870409 22868 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0021c3af757495a903b5fdef0d4085b [sys.catalog]: This master's current role is: LEADER
W20260810 04:10:48.871049 22883 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d0021c3af757495a903b5fdef0d4085b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 04:10:48.871115 22883 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 04:10:48.871251 22882 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 04:10:48.871430 22882 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 04:10:48.872844 22882 catalog_manager.cc:1347] Generated new cluster ID: caf8f8132c764996b6706b3959b892b0
I20260810 04:10:48.872895 22882 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 10 04:10:48 dist-test-slave-8w9v krb5kdc[22776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335048, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260810 04:10:48.880663 22865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:48.868367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.214:44821 (local address 127.0.0.1:35913)
0810 04:10:48.868544 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0810 04:10:48.868554 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 04:10:48.868590 (+ 36us) server_negotiation.cc:407] Connection header received
0810 04:10:48.868710 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:10:48.868718 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:10:48.868775 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:10:48.868926 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:10:48.869686 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:48.871798 (+ 2112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:48.872867 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:48.873454 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:48.873564 (+ 110us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:10:48.876241 (+ 2677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:10:48.876245 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:10:48.876245 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:10:48.876257 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:10:48.878891 (+ 2634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:10:48.879355 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:10:48.879359 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:10:48.879359 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:10:48.879481 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:10:48.879726 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:10:48.879727 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:10:48.879727 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:10:48.880038 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:10:48.880250 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0810 04:10:48.880359 (+ 109us) server_negotiation.cc:299] Negotiation successful
0810 04:10:48.880465 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":95,"threads_started":1}
I20260810 04:10:48.881701 31872 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 04:10:48.890364 22865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:48.883182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.214:41815 (local address 127.0.0.1:35913)
0810 04:10:48.883221 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0810 04:10:48.883223 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:10:48.883360 (+ 137us) server_negotiation.cc:407] Connection header received
0810 04:10:48.883414 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:10:48.883415 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:10:48.883424 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:10:48.883478 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:10:48.884087 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:48.884845 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:48.885916 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:48.886215 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:48.886281 (+ 66us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:10:48.887630 (+ 1349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:10:48.887631 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:10:48.887632 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:10:48.887638 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:10:48.889145 (+ 1507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:10:48.889569 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:10:48.889570 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:10:48.889571 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:10:48.889624 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:10:48.889850 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:10:48.889851 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:10:48.889851 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:10:48.889960 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:10:48.890045 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0810 04:10:48.890212 (+ 167us) server_negotiation.cc:299] Negotiation successful
0810 04:10:48.890281 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260810 04:10:48.896633 22882 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 04:10:48.897327 22882 catalog_manager.cc:1504] Loading token signing keys...
I20260810 04:10:48.900610 22882 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d0021c3af757495a903b5fdef0d4085b: Generated new TSK 0
I20260810 04:10:48.900789 22882 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 04:10:48.906414 31872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskws89Io/build/release/bin/kudu with pid 22789
W20260810 04:10:48.913859 22774 connection.cc:535] client connection to 127.0.0.1:35913 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35913): Connection reset by peer (error 104)
2026-08-10T04:10:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1375 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786334794676589-31872-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 04:10:48 dist-test-slave-8w9v krb5kdc[22896](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 04:10:48 dist-test-slave-8w9v krb5kdc[22896](info): set up 2 sockets
Aug 10 04:10:48 dist-test-slave-8w9v krb5kdc[22896](info): commencing operation
krb5kdc: starting...
W20260810 04:10:49.994805 31872 mini_kdc.cc:121] Time spent starting KDC: real 1.067s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786334794676589-31872-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786334794676589-31872-0/krb5kdc/test-user.keytab.
Aug 10 04:10:50 dist-test-slave-8w9v krb5kdc[22896](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335050, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T04:10:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T04:10:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786334794676589-31872-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786334794676589-31872-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 04:10:50.117106 31872 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskws89Io/build/release/bin/kudu
/tmp/dist-test-taskws89Io/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786334794676589-31872-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786334794676589-31872-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786334794676589-31872-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786334794676589-31872-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40479
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786334794676589-31872-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40479
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786334794676589-31872-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786334794676589-31872-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786334794676589-31872-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:10:50.198982 22914 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:10:50.199187 22914 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:10:50.199220 22914 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:10:50.200400 22914 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:10:50.200443 22914 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:10:50.200456 22914 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:10:50.200515 22914 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:10:50.200541 22914 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:10:50.202041 22914 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786334794676589-31872-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786334794676589-31872-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40479
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786334794676589-31872-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40479
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786334794676589-31872-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22914
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786334794676589-31872-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:10:50.202338 22914 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:10:50.202580 22914 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:10:50.207444 22920 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:10:50.207450 22919 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:10:50.207486 22922 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:10:50.207701 22914 server_base.cc:1034] running on GCE node
I20260810 04:10:50.207979 22914 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:10:50.208276 22914 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:10:50.209420 22914 hybrid_clock.cc:648] HybridClock initialized: now 1786335050209399 us; error 33 us; skew 500 ppm
Aug 10 04:10:50 dist-test-slave-8w9v krb5kdc[22896](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335050, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:10:50.216404 22914 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 04:10:50.216817 22914 webserver.cc:460] Webserver started at http://127.0.0.1:38229/ using document root <none> and password file <none>
I20260810 04:10:50.217060 22914 fs_manager.cc:359] Metadata directory not provided
I20260810 04:10:50.217131 22914 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:10:50.217281 22914 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:10:50.218320 22914 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786334794676589-31872-0/minicluster-data/master-0/data/instance:
uuid: "09339d601816474aa42230695123fde1"
format_stamp: "Formatted at 2026-08-10 04:10:50 on dist-test-slave-8w9v"
server_key: "9e8f8dbae2f444e3218f17fcda24f6e7"
server_key_iv: "a55f8d65a7721df9714e0e81011cd26a"
server_key_version: "encryptionkey@0"
I20260810 04:10:50.218706 22914 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786334794676589-31872-0/minicluster-data/master-0/wal/instance:
uuid: "09339d601816474aa42230695123fde1"
format_stamp: "Formatted at 2026-08-10 04:10:50 on dist-test-slave-8w9v"
server_key: "9e8f8dbae2f444e3218f17fcda24f6e7"
server_key_iv: "a55f8d65a7721df9714e0e81011cd26a"
server_key_version: "encryptionkey@0"
I20260810 04:10:50.220057 22914 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260810 04:10:50.220841 22930 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:10:50.221024 22914 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 04:10:50.221125 22914 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786334794676589-31872-0/minicluster-data/master-0/data,/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786334794676589-31872-0/minicluster-data/master-0/wal
uuid: "09339d601816474aa42230695123fde1"
format_stamp: "Formatted at 2026-08-10 04:10:50 on dist-test-slave-8w9v"
server_key: "9e8f8dbae2f444e3218f17fcda24f6e7"
server_key_iv: "a55f8d65a7721df9714e0e81011cd26a"
server_key_version: "encryptionkey@0"
I20260810 04:10:50.221208 22914 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786334794676589-31872-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786334794676589-31872-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786334794676589-31872-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:10:50.231730 22914 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:10:50.233520 22914 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:10:50.233680 22914 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:10:50.237905 22914 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40479
I20260810 04:10:50.237965 22985 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40479 every 8 connection(s)
I20260810 04:10:50.238299 22914 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786334794676589-31872-0/minicluster-data/master-0/data/info.pb
I20260810 04:10:50.239107 22986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:10:50.241379 22986 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09339d601816474aa42230695123fde1: Bootstrap starting.
I20260810 04:10:50.241765 31872 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskws89Io/build/release/bin/kudu as pid 22914
I20260810 04:10:50.241901 31872 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786334794676589-31872-0/minicluster-data/master-0/wal/instance
I20260810 04:10:50.242125 31872 external_mini_cluster.cc:1428] Setting key b4a5a790c8de6ec90ba53dd6f00edccd
I20260810 04:10:50.242125 22986 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 09339d601816474aa42230695123fde1: Neither blocks nor log segments found. Creating new log.
I20260810 04:10:50.242471 22986 log.cc:824] T 00000000000000000000000000000000 P 09339d601816474aa42230695123fde1: Log is configured to *not* fsync() on all Append() calls
I20260810 04:10:50.243295 22986 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09339d601816474aa42230695123fde1: No bootstrap required, opened a new log
I20260810 04:10:50.245324 22986 raft_consensus.cc:348] T 00000000000000000000000000000000 P 09339d601816474aa42230695123fde1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09339d601816474aa42230695123fde1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40479 } }
I20260810 04:10:50.245528 22986 raft_consensus.cc:374] T 00000000000000000000000000000000 P 09339d601816474aa42230695123fde1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:10:50.245569 22986 raft_consensus.cc:729] T 00000000000000000000000000000000 P 09339d601816474aa42230695123fde1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09339d601816474aa42230695123fde1, State: Initialized, Role: FOLLOWER
I20260810 04:10:50.245785 22986 consensus_queue.cc:260] T 00000000000000000000000000000000 P 09339d601816474aa42230695123fde1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09339d601816474aa42230695123fde1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40479 } }
I20260810 04:10:50.245862 22986 raft_consensus.cc:388] T 00000000000000000000000000000000 P 09339d601816474aa42230695123fde1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 04:10:50.245903 22986 raft_consensus.cc:482] T 00000000000000000000000000000000 P 09339d601816474aa42230695123fde1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 04:10:50.245952 22986 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 09339d601816474aa42230695123fde1 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:10:50.246621 22986 raft_consensus.cc:504] T 00000000000000000000000000000000 P 09339d601816474aa42230695123fde1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09339d601816474aa42230695123fde1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40479 } }
I20260810 04:10:50.246747 22986 leader_election.cc:302] T 00000000000000000000000000000000 P 09339d601816474aa42230695123fde1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 09339d601816474aa42230695123fde1; no voters:
I20260810 04:10:50.246912 22986 leader_election.cc:288] T 00000000000000000000000000000000 P 09339d601816474aa42230695123fde1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 04:10:50.246959 22991 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 09339d601816474aa42230695123fde1 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:10:50.247115 22991 raft_consensus.cc:686] T 00000000000000000000000000000000 P 09339d601816474aa42230695123fde1 [term 1 LEADER]: Becoming Leader. State: Replica: 09339d601816474aa42230695123fde1, State: Running, Role: LEADER
I20260810 04:10:50.247160 22986 sys_catalog.cc:564] T 00000000000000000000000000000000 P 09339d601816474aa42230695123fde1 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:10:50.247200 22991 consensus_queue.cc:237] T 00000000000000000000000000000000 P 09339d601816474aa42230695123fde1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09339d601816474aa42230695123fde1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40479 } }
I20260810 04:10:50.247537 22991 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09339d601816474aa42230695123fde1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 09339d601816474aa42230695123fde1. Latest consensus state: current_term: 1 leader_uuid: "09339d601816474aa42230695123fde1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09339d601816474aa42230695123fde1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40479 } } }
I20260810 04:10:50.247645 22992 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09339d601816474aa42230695123fde1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "09339d601816474aa42230695123fde1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09339d601816474aa42230695123fde1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40479 } } }
I20260810 04:10:50.247666 22991 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09339d601816474aa42230695123fde1 [sys.catalog]: This master's current role is: LEADER
I20260810 04:10:50.247802 22992 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09339d601816474aa42230695123fde1 [sys.catalog]: This master's current role is: LEADER
I20260810 04:10:50.248008 22999 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 04:10:50.248764 22999 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 04:10:50.249884 22990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:50.242648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.214:49807 (local address 127.0.0.1:40479)
0810 04:10:50.242807 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0810 04:10:50.242815 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 04:10:50.242845 (+ 30us) server_negotiation.cc:407] Connection header received
0810 04:10:50.243455 (+ 610us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:10:50.243465 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:10:50.243527 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:10:50.243663 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:10:50.243867 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:50.245918 (+ 2051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:50.247101 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:50.247732 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:50.247846 (+ 114us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:10:50.249083 (+ 1237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:10:50.249087 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:10:50.249087 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:10:50.249304 (+ 217us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:10:50.249343 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:10:50.249357 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:10:50.249450 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0810 04:10:50.249643 (+ 193us) server_negotiation.cc:299] Negotiation successful
0810 04:10:50.249672 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":80,"threads_started":1}
I20260810 04:10:50.250434 22999 catalog_manager.cc:1347] Generated new cluster ID: eb27cdadbb0a4915a08e3e30fa013a56
I20260810 04:10:50.250487 22999 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 04:10:50.250921 31872 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 04:10:50.262485 22999 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 04:10:50.262997 22999 catalog_manager.cc:1504] Loading token signing keys...
I20260810 04:10:50.265609 22990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:50.252338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.214:44115 (local address 127.0.0.1:40479)
0810 04:10:50.252369 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0810 04:10:50.252370 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 04:10:50.262160 (+ 9790us) server_negotiation.cc:407] Connection header received
0810 04:10:50.262188 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:10:50.262189 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:10:50.262200 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:10:50.262255 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:10:50.262467 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:50.263207 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:50.264203 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:50.264556 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:50.264633 (+ 77us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:10:50.265335 (+ 702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:10:50.265336 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:10:50.265337 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:10:50.265343 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:10:50.265360 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:10:50.265364 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:10:50.265436 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0810 04:10:50.265510 (+ 74us) server_negotiation.cc:299] Negotiation successful
0810 04:10:50.265524 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260810 04:10:50.270897 22999 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 09339d601816474aa42230695123fde1: Generated new TSK 0
I20260810 04:10:50.271098 22999 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 04:10:50.273191 31872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskws89Io/build/release/bin/kudu with pid 22914
W20260810 04:10:50.280599 22894 connection.cc:535] client connection to 127.0.0.1:40479 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40479): Connection reset by peer (error 104)
2026-08-10T04:10:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1366 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786334794676589-31872-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 04:10:50 dist-test-slave-8w9v krb5kdc[23020](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 04:10:50 dist-test-slave-8w9v krb5kdc[23020](info): set up 2 sockets
Aug 10 04:10:50 dist-test-slave-8w9v krb5kdc[23020](info): commencing operation
krb5kdc: starting...
W20260810 04:10:51.355908 31872 mini_kdc.cc:121] Time spent starting KDC: real 1.061s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786334794676589-31872-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786334794676589-31872-0/krb5kdc/test-user.keytab.
Aug 10 04:10:51 dist-test-slave-8w9v krb5kdc[23020](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335051, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T04:10:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T04:10:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786334794676589-31872-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786334794676589-31872-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 04:10:51.478992 31872 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskws89Io/build/release/bin/kudu
/tmp/dist-test-taskws89Io/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786334794676589-31872-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786334794676589-31872-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786334794676589-31872-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786334794676589-31872-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34887
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786334794676589-31872-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34887
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786334794676589-31872-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786334794676589-31872-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786334794676589-31872-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:10:51.559712 23035 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:10:51.559916 23035 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:10:51.559940 23035 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:10:51.561028 23035 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:10:51.561064 23035 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:10:51.561077 23035 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:10:51.561089 23035 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:10:51.561101 23035 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:10:51.562436 23035 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786334794676589-31872-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786334794676589-31872-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34887
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786334794676589-31872-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34887
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786334794676589-31872-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23035
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786334794676589-31872-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:10:51.562616 23035 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:10:51.562772 23035 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:10:51.567423 23045 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:10:51.567430 23043 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:10:51.567497 23042 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:10:51.567654 23035 server_base.cc:1034] running on GCE node
I20260810 04:10:51.567934 23035 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:10:51.568276 23035 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:10:51.569448 23035 hybrid_clock.cc:648] HybridClock initialized: now 1786335051569434 us; error 31 us; skew 500 ppm
Aug 10 04:10:51 dist-test-slave-8w9v krb5kdc[23020](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335051, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:10:51.571906 23035 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 04:10:51.572393 23035 webserver.cc:460] Webserver started at http://127.0.0.1:35225/ using document root <none> and password file <none>
I20260810 04:10:51.572618 23035 fs_manager.cc:359] Metadata directory not provided
I20260810 04:10:51.572685 23035 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:10:51.572835 23035 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:10:51.573920 23035 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786334794676589-31872-0/minicluster-data/master-0/data/instance:
uuid: "638336538c6b437db3e5dde5ed56b23a"
format_stamp: "Formatted at 2026-08-10 04:10:51 on dist-test-slave-8w9v"
server_key: "b0abee7c622ae7130bc55d10a90b24b7"
server_key_iv: "079f915b4373d8690b3bfeabec50422b"
server_key_version: "encryptionkey@0"
I20260810 04:10:51.574285 23035 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786334794676589-31872-0/minicluster-data/master-0/wal/instance:
uuid: "638336538c6b437db3e5dde5ed56b23a"
format_stamp: "Formatted at 2026-08-10 04:10:51 on dist-test-slave-8w9v"
server_key: "b0abee7c622ae7130bc55d10a90b24b7"
server_key_iv: "079f915b4373d8690b3bfeabec50422b"
server_key_version: "encryptionkey@0"
I20260810 04:10:51.576038 23035 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260810 04:10:51.576790 23052 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:10:51.576960 23035 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 04:10:51.577062 23035 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786334794676589-31872-0/minicluster-data/master-0/data,/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786334794676589-31872-0/minicluster-data/master-0/wal
uuid: "638336538c6b437db3e5dde5ed56b23a"
format_stamp: "Formatted at 2026-08-10 04:10:51 on dist-test-slave-8w9v"
server_key: "b0abee7c622ae7130bc55d10a90b24b7"
server_key_iv: "079f915b4373d8690b3bfeabec50422b"
server_key_version: "encryptionkey@0"
I20260810 04:10:51.577145 23035 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786334794676589-31872-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786334794676589-31872-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786334794676589-31872-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:10:51.580269 23035 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:10:51.582501 23035 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:10:51.582652 23035 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:10:51.586632 23035 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34887
I20260810 04:10:51.586683 23110 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34887 every 8 connection(s)
I20260810 04:10:51.587002 23035 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786334794676589-31872-0/minicluster-data/master-0/data/info.pb
I20260810 04:10:51.587956 23111 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:10:51.590214 23111 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 638336538c6b437db3e5dde5ed56b23a: Bootstrap starting.
I20260810 04:10:51.590781 23111 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 638336538c6b437db3e5dde5ed56b23a: Neither blocks nor log segments found. Creating new log.
I20260810 04:10:51.591073 23111 log.cc:824] T 00000000000000000000000000000000 P 638336538c6b437db3e5dde5ed56b23a: Log is configured to *not* fsync() on all Append() calls
I20260810 04:10:51.591782 23111 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 638336538c6b437db3e5dde5ed56b23a: No bootstrap required, opened a new log
I20260810 04:10:51.593434 23111 raft_consensus.cc:348] T 00000000000000000000000000000000 P 638336538c6b437db3e5dde5ed56b23a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "638336538c6b437db3e5dde5ed56b23a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34887 } }
I20260810 04:10:51.593587 23111 raft_consensus.cc:374] T 00000000000000000000000000000000 P 638336538c6b437db3e5dde5ed56b23a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:10:51.593612 23111 raft_consensus.cc:729] T 00000000000000000000000000000000 P 638336538c6b437db3e5dde5ed56b23a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 638336538c6b437db3e5dde5ed56b23a, State: Initialized, Role: FOLLOWER
I20260810 04:10:51.593835 31872 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskws89Io/build/release/bin/kudu as pid 23035
I20260810 04:10:51.593852 23111 consensus_queue.cc:260] T 00000000000000000000000000000000 P 638336538c6b437db3e5dde5ed56b23a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "638336538c6b437db3e5dde5ed56b23a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34887 } }
I20260810 04:10:51.593943 23111 raft_consensus.cc:388] T 00000000000000000000000000000000 P 638336538c6b437db3e5dde5ed56b23a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 04:10:51.593961 31872 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786334794676589-31872-0/minicluster-data/master-0/wal/instance
I20260810 04:10:51.593987 23111 raft_consensus.cc:482] T 00000000000000000000000000000000 P 638336538c6b437db3e5dde5ed56b23a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 04:10:51.594041 23111 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 638336538c6b437db3e5dde5ed56b23a [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:10:51.594189 31872 external_mini_cluster.cc:1428] Setting key 9a81c4564800cd3921ef773a83210e9d
I20260810 04:10:51.594738 23111 raft_consensus.cc:504] T 00000000000000000000000000000000 P 638336538c6b437db3e5dde5ed56b23a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "638336538c6b437db3e5dde5ed56b23a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34887 } }
I20260810 04:10:51.594870 23111 leader_election.cc:302] T 00000000000000000000000000000000 P 638336538c6b437db3e5dde5ed56b23a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 638336538c6b437db3e5dde5ed56b23a; no voters:
I20260810 04:10:51.595050 23111 leader_election.cc:288] T 00000000000000000000000000000000 P 638336538c6b437db3e5dde5ed56b23a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 04:10:51.595074 23116 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 638336538c6b437db3e5dde5ed56b23a [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:10:51.595182 23116 raft_consensus.cc:686] T 00000000000000000000000000000000 P 638336538c6b437db3e5dde5ed56b23a [term 1 LEADER]: Becoming Leader. State: Replica: 638336538c6b437db3e5dde5ed56b23a, State: Running, Role: LEADER
I20260810 04:10:51.595314 23111 sys_catalog.cc:564] T 00000000000000000000000000000000 P 638336538c6b437db3e5dde5ed56b23a [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:10:51.595346 23116 consensus_queue.cc:237] T 00000000000000000000000000000000 P 638336538c6b437db3e5dde5ed56b23a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "638336538c6b437db3e5dde5ed56b23a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34887 } }
I20260810 04:10:51.595839 23117 sys_catalog.cc:455] T 00000000000000000000000000000000 P 638336538c6b437db3e5dde5ed56b23a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "638336538c6b437db3e5dde5ed56b23a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "638336538c6b437db3e5dde5ed56b23a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34887 } } }
I20260810 04:10:51.595942 23117 sys_catalog.cc:458] T 00000000000000000000000000000000 P 638336538c6b437db3e5dde5ed56b23a [sys.catalog]: This master's current role is: LEADER
I20260810 04:10:51.596230 23118 sys_catalog.cc:455] T 00000000000000000000000000000000 P 638336538c6b437db3e5dde5ed56b23a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 638336538c6b437db3e5dde5ed56b23a. Latest consensus state: current_term: 1 leader_uuid: "638336538c6b437db3e5dde5ed56b23a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "638336538c6b437db3e5dde5ed56b23a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34887 } } }
I20260810 04:10:51.596339 23118 sys_catalog.cc:458] T 00000000000000000000000000000000 P 638336538c6b437db3e5dde5ed56b23a [sys.catalog]: This master's current role is: LEADER
I20260810 04:10:51.596737 23120 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 04:10:51.596911 23115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:51.594656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.214:36901 (local address 127.0.0.1:34887)
0810 04:10:51.594845 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0810 04:10:51.594855 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 04:10:51.594888 (+ 33us) server_negotiation.cc:407] Connection header received
0810 04:10:51.595004 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:10:51.595010 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:10:51.595044 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:10:51.595428 (+ 384us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:10:51.596202 (+ 774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:10:51.596206 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:10:51.596206 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:10:51.596211 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:10:51.596259 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:10:51.596276 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:10:51.596361 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0810 04:10:51.596604 (+ 243us) server_negotiation.cc:299] Negotiation successful
0810 04:10:51.596658 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":79,"threads_started":1}
I20260810 04:10:51.597290 23120 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 04:10:51.597947 31872 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 04:10:51.600529 23115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:51.599438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.214:60021 (local address 127.0.0.1:34887)
0810 04:10:51.599491 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0810 04:10:51.599493 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:10:51.599591 (+ 98us) server_negotiation.cc:407] Connection header received
0810 04:10:51.599671 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:10:51.599672 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:10:51.599682 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:10:51.599725 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:10:51.600269 (+ 544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:10:51.600270 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:10:51.600270 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:10:51.600274 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:10:51.600286 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:10:51.600289 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:10:51.600361 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0810 04:10:51.600417 (+ 56us) server_negotiation.cc:299] Negotiation successful
0810 04:10:51.600427 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260810 04:10:51.609161 23120 catalog_manager.cc:1347] Generated new cluster ID: 8ee6598a4c2d4f47a6680a6009ef2b88
I20260810 04:10:51.609223 23120 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 04:10:51.629266 23120 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 04:10:51.629446 23120 catalog_manager.cc:1504] Loading token signing keys...
I20260810 04:10:51.633903 23120 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 638336538c6b437db3e5dde5ed56b23a: Generated new TSK 0
I20260810 04:10:51.634078 23120 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 04:10:51.643502 31872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskws89Io/build/release/bin/kudu with pid 23035
2026-08-10T04:10:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1368 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786334794676589-31872-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 04:10:51 dist-test-slave-8w9v krb5kdc[23143](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 04:10:51 dist-test-slave-8w9v krb5kdc[23143](info): set up 2 sockets
Aug 10 04:10:51 dist-test-slave-8w9v krb5kdc[23143](info): commencing operation
krb5kdc: starting...
W20260810 04:10:52.740948 31872 mini_kdc.cc:121] Time spent starting KDC: real 1.078s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786334794676589-31872-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786334794676589-31872-0/krb5kdc/test-user.keytab.
Aug 10 04:10:52 dist-test-slave-8w9v krb5kdc[23143](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335052, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T04:10:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T04:10:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.32.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.32.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.32.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.62.keytab.
Entry for principal kudu/127.31.32.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.62.keytab.
I20260810 04:10:52.860934 31872 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskws89Io/build/release/bin/kudu
/tmp/dist-test-taskws89Io/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786334794676589-31872-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786334794676589-31872-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786334794676589-31872-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786334794676589-31872-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.32.62:33341
--webserver_interface=127.31.32.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.62.keytab
--principal=kudu/127.31.32.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.32.20:35745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.32.62:33341
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786334794676589-31872-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786334794676589-31872-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786334794676589-31872-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:10:52.942648 23158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:10:52.942884 23158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:10:52.942924 23158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:10:52.944056 23158 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:10:52.944121 23158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:10:52.944164 23158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:10:52.944196 23158 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:10:52.944223 23158 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:10:52.945695 23158 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.32.20:35745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786334794676589-31872-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786334794676589-31872-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.32.62:33341
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.62.keytab
--principal=kudu/127.31.32.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.32.62:33341
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786334794676589-31872-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.32.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786334794676589-31872-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:10:52.945974 23158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:10:52.946205 23158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:10:52.951224 23164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:10:52.951247 23167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:10:52.951587 23158 server_base.cc:1034] running on GCE node
W20260810 04:10:52.951282 23165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:10:52.951879 23158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:10:52.952230 23158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:10:52.953425 23158 hybrid_clock.cc:648] HybridClock initialized: now 1786335052953407 us; error 46 us; skew 500 ppm
Aug 10 04:10:52 dist-test-slave-8w9v krb5kdc[23143](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335052, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:10:52.955876 23158 init.cc:377] Logged in from keytab as kudu/127.31.32.62@KRBTEST.COM (short username kudu)
I20260810 04:10:52.956369 23158 webserver.cc:460] Webserver started at http://127.31.32.62:40777/ using document root <none> and password file <none>
I20260810 04:10:52.956598 23158 fs_manager.cc:359] Metadata directory not provided
I20260810 04:10:52.956650 23158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:10:52.956822 23158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:10:52.957880 23158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786334794676589-31872-0/minicluster-data/master-0/data/instance:
uuid: "dfd61712789f40fe91fe8f6538b623ce"
format_stamp: "Formatted at 2026-08-10 04:10:52 on dist-test-slave-8w9v"
server_key: "37dde323233dc4a8fd793361ae0e2419"
server_key_iv: "136215e622010bdee23ecd4c43e723fc"
server_key_version: "encryptionkey@0"
I20260810 04:10:52.958300 23158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786334794676589-31872-0/minicluster-data/master-0/wal/instance:
uuid: "dfd61712789f40fe91fe8f6538b623ce"
format_stamp: "Formatted at 2026-08-10 04:10:52 on dist-test-slave-8w9v"
server_key: "37dde323233dc4a8fd793361ae0e2419"
server_key_iv: "136215e622010bdee23ecd4c43e723fc"
server_key_version: "encryptionkey@0"
I20260810 04:10:52.959692 23158 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260810 04:10:52.960544 23174 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:10:52.960748 23158 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260810 04:10:52.960824 23158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786334794676589-31872-0/minicluster-data/master-0/data,/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786334794676589-31872-0/minicluster-data/master-0/wal
uuid: "dfd61712789f40fe91fe8f6538b623ce"
format_stamp: "Formatted at 2026-08-10 04:10:52 on dist-test-slave-8w9v"
server_key: "37dde323233dc4a8fd793361ae0e2419"
server_key_iv: "136215e622010bdee23ecd4c43e723fc"
server_key_version: "encryptionkey@0"
I20260810 04:10:52.960918 23158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786334794676589-31872-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786334794676589-31872-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786334794676589-31872-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:10:52.992420 23158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:10:52.994566 23158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:10:52.994748 23158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:10:52.998997 23158 rpc_server.cc:307] RPC server started. Bound to: 127.31.32.62:33341
I20260810 04:10:52.999058 23234 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.32.62:33341 every 8 connection(s)
I20260810 04:10:52.999439 23158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786334794676589-31872-0/minicluster-data/master-0/data/info.pb
I20260810 04:10:53.000408 23236 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:10:53.002764 23236 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dfd61712789f40fe91fe8f6538b623ce: Bootstrap starting.
I20260810 04:10:53.003360 23236 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dfd61712789f40fe91fe8f6538b623ce: Neither blocks nor log segments found. Creating new log.
I20260810 04:10:53.003607 23236 log.cc:824] T 00000000000000000000000000000000 P dfd61712789f40fe91fe8f6538b623ce: Log is configured to *not* fsync() on all Append() calls
I20260810 04:10:53.004388 23236 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dfd61712789f40fe91fe8f6538b623ce: No bootstrap required, opened a new log
I20260810 04:10:53.005929 23236 raft_consensus.cc:348] T 00000000000000000000000000000000 P dfd61712789f40fe91fe8f6538b623ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfd61712789f40fe91fe8f6538b623ce" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 33341 } }
I20260810 04:10:53.006069 23236 raft_consensus.cc:374] T 00000000000000000000000000000000 P dfd61712789f40fe91fe8f6538b623ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:10:53.006091 23236 raft_consensus.cc:729] T 00000000000000000000000000000000 P dfd61712789f40fe91fe8f6538b623ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfd61712789f40fe91fe8f6538b623ce, State: Initialized, Role: FOLLOWER
I20260810 04:10:53.006196 23236 consensus_queue.cc:260] T 00000000000000000000000000000000 P dfd61712789f40fe91fe8f6538b623ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfd61712789f40fe91fe8f6538b623ce" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 33341 } }
I20260810 04:10:53.006299 23236 raft_consensus.cc:388] T 00000000000000000000000000000000 P dfd61712789f40fe91fe8f6538b623ce [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 04:10:53.006345 23236 raft_consensus.cc:482] T 00000000000000000000000000000000 P dfd61712789f40fe91fe8f6538b623ce [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 04:10:53.006395 23236 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dfd61712789f40fe91fe8f6538b623ce [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:10:53.006578 31872 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskws89Io/build/release/bin/kudu as pid 23158
I20260810 04:10:53.006711 31872 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786334794676589-31872-0/minicluster-data/master-0/wal/instance
I20260810 04:10:53.006947 31872 external_mini_cluster.cc:1428] Setting key 1df7c9090917ee82d753194b84240e33
I20260810 04:10:53.007571 23236 raft_consensus.cc:504] T 00000000000000000000000000000000 P dfd61712789f40fe91fe8f6538b623ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfd61712789f40fe91fe8f6538b623ce" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 33341 } }
I20260810 04:10:53.007688 23236 leader_election.cc:302] T 00000000000000000000000000000000 P dfd61712789f40fe91fe8f6538b623ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dfd61712789f40fe91fe8f6538b623ce; no voters:
I20260810 04:10:53.007862 23236 leader_election.cc:288] T 00000000000000000000000000000000 P dfd61712789f40fe91fe8f6538b623ce [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 04:10:53.007903 23241 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dfd61712789f40fe91fe8f6538b623ce [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:10:53.008011 23241 raft_consensus.cc:686] T 00000000000000000000000000000000 P dfd61712789f40fe91fe8f6538b623ce [term 1 LEADER]: Becoming Leader. State: Replica: dfd61712789f40fe91fe8f6538b623ce, State: Running, Role: LEADER
I20260810 04:10:53.008172 23236 sys_catalog.cc:564] T 00000000000000000000000000000000 P dfd61712789f40fe91fe8f6538b623ce [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:10:53.008210 23241 consensus_queue.cc:237] T 00000000000000000000000000000000 P dfd61712789f40fe91fe8f6538b623ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfd61712789f40fe91fe8f6538b623ce" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 33341 } }
I20260810 04:10:53.008708 23243 sys_catalog.cc:455] T 00000000000000000000000000000000 P dfd61712789f40fe91fe8f6538b623ce [sys.catalog]: SysCatalogTable state changed. Reason: New leader dfd61712789f40fe91fe8f6538b623ce. Latest consensus state: current_term: 1 leader_uuid: "dfd61712789f40fe91fe8f6538b623ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfd61712789f40fe91fe8f6538b623ce" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 33341 } } }
I20260810 04:10:53.008827 23243 sys_catalog.cc:458] T 00000000000000000000000000000000 P dfd61712789f40fe91fe8f6538b623ce [sys.catalog]: This master's current role is: LEADER
I20260810 04:10:53.009241 23242 sys_catalog.cc:455] T 00000000000000000000000000000000 P dfd61712789f40fe91fe8f6538b623ce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dfd61712789f40fe91fe8f6538b623ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfd61712789f40fe91fe8f6538b623ce" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 33341 } } }
I20260810 04:10:53.009332 23242 sys_catalog.cc:458] T 00000000000000000000000000000000 P dfd61712789f40fe91fe8f6538b623ce [sys.catalog]: This master's current role is: LEADER
W20260810 04:10:53.010087 23258 catalog_manager.cc:1558] T 00000000000000000000000000000000 P dfd61712789f40fe91fe8f6538b623ce: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 04:10:53.010195 23258 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 04:10:53.010305 23252 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 04:10:53.010496 23252 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 04:10:53.012355 23252 catalog_manager.cc:1347] Generated new cluster ID: 5393b9e887f94ef1b434ef04d69f7820
I20260810 04:10:53.012416 23252 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 10 04:10:53 dist-test-slave-8w9v krb5kdc[23143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335052, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.32.62@KRBTEST.COM
I20260810 04:10:53.019351 23240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:53.007518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37943 (local address 127.31.32.62:33341)
0810 04:10:53.007774 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0810 04:10:53.007786 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 04:10:53.007822 (+ 36us) server_negotiation.cc:407] Connection header received
0810 04:10:53.007984 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:10:53.007992 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:10:53.008056 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:10:53.009289 (+ 1233us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:10:53.009418 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:53.010938 (+ 1520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:53.012073 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:53.012703 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:53.012859 (+ 156us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:10:53.015506 (+ 2647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:10:53.015511 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:10:53.015511 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:10:53.015523 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:10:53.017682 (+ 2159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:10:53.018102 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:10:53.018106 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:10:53.018106 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:10:53.018221 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:10:53.018477 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:10:53.018478 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:10:53.018479 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:10:53.018732 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:10:53.018930 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0810 04:10:53.019025 (+ 95us) server_negotiation.cc:299] Negotiation successful
0810 04:10:53.019135 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":107,"threads_started":1}
I20260810 04:10:53.020519 31872 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 04:10:53.025939 23252 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 04:10:53.026602 23252 catalog_manager.cc:1504] Loading token signing keys...
I20260810 04:10:53.028198 23240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:53.021933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51335 (local address 127.31.32.62:33341)
0810 04:10:53.021972 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0810 04:10:53.021974 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:10:53.022091 (+ 117us) server_negotiation.cc:407] Connection header received
0810 04:10:53.022164 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:10:53.022165 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:10:53.022175 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:10:53.022231 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:10:53.022795 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:53.023297 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:53.023989 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:53.024280 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:53.024377 (+ 97us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:10:53.025466 (+ 1089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:10:53.025468 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:10:53.025468 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:10:53.025475 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:10:53.026991 (+ 1516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:10:53.027367 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:10:53.027369 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:10:53.027369 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:10:53.027423 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:10:53.027660 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:10:53.027661 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:10:53.027661 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:10:53.027768 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:10:53.027877 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0810 04:10:53.028003 (+ 126us) server_negotiation.cc:299] Negotiation successful
0810 04:10:53.028065 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260810 04:10:53.039211 23252 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dfd61712789f40fe91fe8f6538b623ce: Generated new TSK 0
I20260810 04:10:53.039453 23252 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 04:10:53.043368 31872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskws89Io/build/release/bin/kudu with pid 23158
W20260810 04:10:53.050434 23141 connection.cc:535] client connection to 127.31.32.62:33341 recv error: Network error: failed to read from TLS socket (remote: 127.31.32.62:33341): Connection reset by peer (error 104)
2026-08-10T04:10:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1401 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786334794676589-31872-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 04:10:53 dist-test-slave-8w9v krb5kdc[23269](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 04:10:53 dist-test-slave-8w9v krb5kdc[23269](info): set up 2 sockets
Aug 10 04:10:53 dist-test-slave-8w9v krb5kdc[23269](info): commencing operation
krb5kdc: starting...
W20260810 04:10:54.146188 31872 mini_kdc.cc:121] Time spent starting KDC: real 1.082s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786334794676589-31872-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786334794676589-31872-0/krb5kdc/test-user.keytab.
Aug 10 04:10:54 dist-test-slave-8w9v krb5kdc[23269](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335054, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T04:10:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T04:10:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.32.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.32.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.32.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.62.keytab.
Entry for principal kudu/127.31.32.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.62.keytab.
I20260810 04:10:54.271162 31872 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskws89Io/build/release/bin/kudu
/tmp/dist-test-taskws89Io/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786334794676589-31872-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786334794676589-31872-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786334794676589-31872-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786334794676589-31872-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.32.62:40569
--webserver_interface=127.31.32.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.62.keytab
--principal=kudu/127.31.32.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.32.20:39581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.32.62:40569
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786334794676589-31872-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786334794676589-31872-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786334794676589-31872-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:10:54.353494 23283 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:10:54.353747 23283 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:10:54.353781 23283 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:10:54.354914 23283 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:10:54.354984 23283 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:10:54.355008 23283 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:10:54.355057 23283 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:10:54.355086 23283 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:10:54.356630 23283 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.32.20:39581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786334794676589-31872-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786334794676589-31872-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.32.62:40569
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.62.keytab
--principal=kudu/127.31.32.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.32.62:40569
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786334794676589-31872-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.32.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23283
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786334794676589-31872-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:10:54.356896 23283 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:10:54.357124 23283 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:10:54.362035 23291 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:10:54.362098 23289 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:10:54.362037 23288 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:10:54.362118 23283 server_base.cc:1034] running on GCE node
I20260810 04:10:54.362574 23283 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:10:54.362876 23283 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:10:54.364046 23283 hybrid_clock.cc:648] HybridClock initialized: now 1786335054364040 us; error 40 us; skew 500 ppm
Aug 10 04:10:54 dist-test-slave-8w9v krb5kdc[23269](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335054, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:10:54.366545 23283 init.cc:377] Logged in from keytab as kudu/127.31.32.62@KRBTEST.COM (short username kudu)
I20260810 04:10:54.367002 23283 webserver.cc:460] Webserver started at http://127.31.32.62:37641/ using document root <none> and password file <none>
I20260810 04:10:54.367250 23283 fs_manager.cc:359] Metadata directory not provided
I20260810 04:10:54.367326 23283 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:10:54.367478 23283 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:10:54.368613 23283 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786334794676589-31872-0/minicluster-data/master-0/data/instance:
uuid: "f7bb784907224e4aa7403b86eb20b23f"
format_stamp: "Formatted at 2026-08-10 04:10:54 on dist-test-slave-8w9v"
server_key: "f1151853ed5444c59e77a9937e3bd517"
server_key_iv: "8917d31aee79b316dab090ba84c6d921"
server_key_version: "encryptionkey@0"
I20260810 04:10:54.368988 23283 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786334794676589-31872-0/minicluster-data/master-0/wal/instance:
uuid: "f7bb784907224e4aa7403b86eb20b23f"
format_stamp: "Formatted at 2026-08-10 04:10:54 on dist-test-slave-8w9v"
server_key: "f1151853ed5444c59e77a9937e3bd517"
server_key_iv: "8917d31aee79b316dab090ba84c6d921"
server_key_version: "encryptionkey@0"
I20260810 04:10:54.370323 23283 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 04:10:54.371083 23299 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:10:54.371294 23283 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:10:54.371390 23283 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786334794676589-31872-0/minicluster-data/master-0/data,/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786334794676589-31872-0/minicluster-data/master-0/wal
uuid: "f7bb784907224e4aa7403b86eb20b23f"
format_stamp: "Formatted at 2026-08-10 04:10:54 on dist-test-slave-8w9v"
server_key: "f1151853ed5444c59e77a9937e3bd517"
server_key_iv: "8917d31aee79b316dab090ba84c6d921"
server_key_version: "encryptionkey@0"
I20260810 04:10:54.371474 23283 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786334794676589-31872-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786334794676589-31872-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786334794676589-31872-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:10:54.395476 23283 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:10:54.397511 23283 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:10:54.397676 23283 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:10:54.401744 23283 rpc_server.cc:307] RPC server started. Bound to: 127.31.32.62:40569
I20260810 04:10:54.401785 23354 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.32.62:40569 every 8 connection(s)
I20260810 04:10:54.402179 23283 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786334794676589-31872-0/minicluster-data/master-0/data/info.pb
I20260810 04:10:54.403146 23355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:10:54.405541 23355 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7bb784907224e4aa7403b86eb20b23f: Bootstrap starting.
I20260810 04:10:54.406140 23355 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f7bb784907224e4aa7403b86eb20b23f: Neither blocks nor log segments found. Creating new log.
I20260810 04:10:54.406436 23355 log.cc:824] T 00000000000000000000000000000000 P f7bb784907224e4aa7403b86eb20b23f: Log is configured to *not* fsync() on all Append() calls
I20260810 04:10:54.406721 31872 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskws89Io/build/release/bin/kudu as pid 23283
I20260810 04:10:54.406852 31872 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786334794676589-31872-0/minicluster-data/master-0/wal/instance
I20260810 04:10:54.407073 31872 external_mini_cluster.cc:1428] Setting key db3f3279c77e6eefb45d83b95411ff3d
I20260810 04:10:54.407272 23355 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7bb784907224e4aa7403b86eb20b23f: No bootstrap required, opened a new log
I20260810 04:10:54.410295 23355 raft_consensus.cc:348] T 00000000000000000000000000000000 P f7bb784907224e4aa7403b86eb20b23f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7bb784907224e4aa7403b86eb20b23f" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 40569 } }
I20260810 04:10:54.410508 23355 raft_consensus.cc:374] T 00000000000000000000000000000000 P f7bb784907224e4aa7403b86eb20b23f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:10:54.410542 23355 raft_consensus.cc:729] T 00000000000000000000000000000000 P f7bb784907224e4aa7403b86eb20b23f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7bb784907224e4aa7403b86eb20b23f, State: Initialized, Role: FOLLOWER
I20260810 04:10:54.410667 23355 consensus_queue.cc:260] T 00000000000000000000000000000000 P f7bb784907224e4aa7403b86eb20b23f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7bb784907224e4aa7403b86eb20b23f" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 40569 } }
I20260810 04:10:54.410755 23355 raft_consensus.cc:388] T 00000000000000000000000000000000 P f7bb784907224e4aa7403b86eb20b23f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 04:10:54.410800 23355 raft_consensus.cc:482] T 00000000000000000000000000000000 P f7bb784907224e4aa7403b86eb20b23f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 04:10:54.410859 23355 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f7bb784907224e4aa7403b86eb20b23f [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:10:54.411427 23355 raft_consensus.cc:504] T 00000000000000000000000000000000 P f7bb784907224e4aa7403b86eb20b23f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7bb784907224e4aa7403b86eb20b23f" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 40569 } }
I20260810 04:10:54.411559 23355 leader_election.cc:302] T 00000000000000000000000000000000 P f7bb784907224e4aa7403b86eb20b23f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f7bb784907224e4aa7403b86eb20b23f; no voters:
I20260810 04:10:54.411720 23355 leader_election.cc:288] T 00000000000000000000000000000000 P f7bb784907224e4aa7403b86eb20b23f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 04:10:54.411767 23360 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f7bb784907224e4aa7403b86eb20b23f [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:10:54.411938 23360 raft_consensus.cc:686] T 00000000000000000000000000000000 P f7bb784907224e4aa7403b86eb20b23f [term 1 LEADER]: Becoming Leader. State: Replica: f7bb784907224e4aa7403b86eb20b23f, State: Running, Role: LEADER
I20260810 04:10:54.411981 23355 sys_catalog.cc:564] T 00000000000000000000000000000000 P f7bb784907224e4aa7403b86eb20b23f [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:10:54.412101 23360 consensus_queue.cc:237] T 00000000000000000000000000000000 P f7bb784907224e4aa7403b86eb20b23f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7bb784907224e4aa7403b86eb20b23f" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 40569 } }
I20260810 04:10:54.412544 23361 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7bb784907224e4aa7403b86eb20b23f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f7bb784907224e4aa7403b86eb20b23f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7bb784907224e4aa7403b86eb20b23f" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 40569 } } }
I20260810 04:10:54.412631 23361 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7bb784907224e4aa7403b86eb20b23f [sys.catalog]: This master's current role is: LEADER
I20260810 04:10:54.412850 23362 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7bb784907224e4aa7403b86eb20b23f [sys.catalog]: SysCatalogTable state changed. Reason: New leader f7bb784907224e4aa7403b86eb20b23f. Latest consensus state: current_term: 1 leader_uuid: "f7bb784907224e4aa7403b86eb20b23f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7bb784907224e4aa7403b86eb20b23f" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 40569 } } }
I20260810 04:10:54.412986 23362 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7bb784907224e4aa7403b86eb20b23f [sys.catalog]: This master's current role is: LEADER
I20260810 04:10:54.413151 23368 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 04:10:54.413694 23368 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 04:10:54.413913 23359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:54.407621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51837 (local address 127.31.32.62:40569)
0810 04:10:54.407824 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0810 04:10:54.407834 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 04:10:54.407874 (+ 40us) server_negotiation.cc:407] Connection header received
0810 04:10:54.408764 (+ 890us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:10:54.408774 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:10:54.408838 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:10:54.408980 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:10:54.409143 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:54.410590 (+ 1447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:54.411513 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:54.412159 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:54.412643 (+ 484us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:10:54.413089 (+ 446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:10:54.413094 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:10:54.413094 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:10:54.413325 (+ 231us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:10:54.413388 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:10:54.413404 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:10:54.413490 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0810 04:10:54.413635 (+ 145us) server_negotiation.cc:299] Negotiation successful
0810 04:10:54.413664 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":91,"threads_started":1}
I20260810 04:10:54.415057 31872 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 04:10:54.416601 23368 catalog_manager.cc:1347] Generated new cluster ID: 45015925260d473bbe465905c53a392c
I20260810 04:10:54.416659 23368 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 04:10:54.419638 23359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:54.416648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33579 (local address 127.31.32.62:40569)
0810 04:10:54.416686 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0810 04:10:54.416688 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:10:54.416871 (+ 183us) server_negotiation.cc:407] Connection header received
0810 04:10:54.416926 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:10:54.416927 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:10:54.416937 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:10:54.416990 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:10:54.417102 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:54.417537 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:54.418480 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:54.418666 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:54.418714 (+ 48us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:10:54.419419 (+ 705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:10:54.419420 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:10:54.419420 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:10:54.419424 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:10:54.419438 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:10:54.419442 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:10:54.419500 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0810 04:10:54.419582 (+ 82us) server_negotiation.cc:299] Negotiation successful
0810 04:10:54.419591 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260810 04:10:54.425812 23368 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 04:10:54.426661 23368 catalog_manager.cc:1504] Loading token signing keys...
I20260810 04:10:54.430788 23368 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f7bb784907224e4aa7403b86eb20b23f: Generated new TSK 0
I20260810 04:10:54.430956 23368 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 04:10:54.436565 31872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskws89Io/build/release/bin/kudu with pid 23283
W20260810 04:10:54.447965 23267 connection.cc:535] client connection to 127.31.32.62:40569 recv error: Network error: failed to read from TLS socket (remote: 127.31.32.62:40569): Connection reset by peer (error 104)
2026-08-10T04:10:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1397 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786334794676589-31872-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 04:10:54 dist-test-slave-8w9v krb5kdc[23388](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 04:10:54 dist-test-slave-8w9v krb5kdc[23388](info): set up 2 sockets
Aug 10 04:10:54 dist-test-slave-8w9v krb5kdc[23388](info): commencing operation
krb5kdc: starting...
W20260810 04:10:55.521178 31872 mini_kdc.cc:121] Time spent starting KDC: real 1.059s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786334794676589-31872-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786334794676589-31872-0/krb5kdc/test-user.keytab.
Aug 10 04:10:55 dist-test-slave-8w9v krb5kdc[23388](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335055, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T04:10:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T04:10:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.32.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.32.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.32.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.62.keytab.
Entry for principal kudu/127.31.32.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.62.keytab.
I20260810 04:10:55.650347 31872 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskws89Io/build/release/bin/kudu
/tmp/dist-test-taskws89Io/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786334794676589-31872-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786334794676589-31872-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786334794676589-31872-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786334794676589-31872-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.32.62:43951
--webserver_interface=127.31.32.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.62.keytab
--principal=kudu/127.31.32.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.32.20:38539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.32.62:43951
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786334794676589-31872-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786334794676589-31872-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786334794676589-31872-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:10:55.732853 23404 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:10:55.733059 23404 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:10:55.733083 23404 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:10:55.734151 23404 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:10:55.734185 23404 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:10:55.734200 23404 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:10:55.734211 23404 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:10:55.734223 23404 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:10:55.735749 23404 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.32.20:38539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786334794676589-31872-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786334794676589-31872-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.32.62:43951
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.62.keytab
--principal=kudu/127.31.32.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.32.62:43951
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786334794676589-31872-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.32.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23404
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786334794676589-31872-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:10:55.735948 23404 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:10:55.736392 23404 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:10:55.741312 23411 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:10:55.741375 23413 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:10:55.741335 23410 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:10:55.741519 23404 server_base.cc:1034] running on GCE node
I20260810 04:10:55.741746 23404 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:10:55.742053 23404 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:10:55.743209 23404 hybrid_clock.cc:648] HybridClock initialized: now 1786335055743199 us; error 31 us; skew 500 ppm
Aug 10 04:10:55 dist-test-slave-8w9v krb5kdc[23388](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335055, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:10:55.750178 23404 init.cc:377] Logged in from keytab as kudu/127.31.32.62@KRBTEST.COM (short username kudu)
I20260810 04:10:55.750763 23404 webserver.cc:460] Webserver started at http://127.31.32.62:42729/ using document root <none> and password file <none>
I20260810 04:10:55.751029 23404 fs_manager.cc:359] Metadata directory not provided
I20260810 04:10:55.751101 23404 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:10:55.751241 23404 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:10:55.752529 23404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786334794676589-31872-0/minicluster-data/master-0/data/instance:
uuid: "cae51875629f4ff0ae751c8ffd16819f"
format_stamp: "Formatted at 2026-08-10 04:10:55 on dist-test-slave-8w9v"
server_key: "fbebf55d75909b0d42f419850f1d5f6f"
server_key_iv: "6026a3e31f7dc39c5974ebbbba2c3444"
server_key_version: "encryptionkey@0"
I20260810 04:10:55.752898 23404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786334794676589-31872-0/minicluster-data/master-0/wal/instance:
uuid: "cae51875629f4ff0ae751c8ffd16819f"
format_stamp: "Formatted at 2026-08-10 04:10:55 on dist-test-slave-8w9v"
server_key: "fbebf55d75909b0d42f419850f1d5f6f"
server_key_iv: "6026a3e31f7dc39c5974ebbbba2c3444"
server_key_version: "encryptionkey@0"
I20260810 04:10:55.754189 23404 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 04:10:55.754885 23421 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:10:55.755048 23404 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 04:10:55.755152 23404 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786334794676589-31872-0/minicluster-data/master-0/data,/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786334794676589-31872-0/minicluster-data/master-0/wal
uuid: "cae51875629f4ff0ae751c8ffd16819f"
format_stamp: "Formatted at 2026-08-10 04:10:55 on dist-test-slave-8w9v"
server_key: "fbebf55d75909b0d42f419850f1d5f6f"
server_key_iv: "6026a3e31f7dc39c5974ebbbba2c3444"
server_key_version: "encryptionkey@0"
I20260810 04:10:55.755226 23404 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786334794676589-31872-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786334794676589-31872-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786334794676589-31872-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:10:55.758579 23404 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:10:55.760509 23404 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:10:55.760656 23404 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:10:55.764955 23404 rpc_server.cc:307] RPC server started. Bound to: 127.31.32.62:43951
I20260810 04:10:55.764986 23480 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.32.62:43951 every 8 connection(s)
I20260810 04:10:55.765316 23404 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786334794676589-31872-0/minicluster-data/master-0/data/info.pb
I20260810 04:10:55.766312 23481 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:10:55.768639 23481 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cae51875629f4ff0ae751c8ffd16819f: Bootstrap starting.
I20260810 04:10:55.769210 23481 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cae51875629f4ff0ae751c8ffd16819f: Neither blocks nor log segments found. Creating new log.
I20260810 04:10:55.769490 23481 log.cc:824] T 00000000000000000000000000000000 P cae51875629f4ff0ae751c8ffd16819f: Log is configured to *not* fsync() on all Append() calls
I20260810 04:10:55.770153 23481 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cae51875629f4ff0ae751c8ffd16819f: No bootstrap required, opened a new log
I20260810 04:10:55.771721 23481 raft_consensus.cc:348] T 00000000000000000000000000000000 P cae51875629f4ff0ae751c8ffd16819f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cae51875629f4ff0ae751c8ffd16819f" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 43951 } }
I20260810 04:10:55.771872 23481 raft_consensus.cc:374] T 00000000000000000000000000000000 P cae51875629f4ff0ae751c8ffd16819f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:10:55.771920 23481 raft_consensus.cc:729] T 00000000000000000000000000000000 P cae51875629f4ff0ae751c8ffd16819f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cae51875629f4ff0ae751c8ffd16819f, State: Initialized, Role: FOLLOWER
I20260810 04:10:55.772109 23481 consensus_queue.cc:260] T 00000000000000000000000000000000 P cae51875629f4ff0ae751c8ffd16819f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cae51875629f4ff0ae751c8ffd16819f" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 43951 } }
I20260810 04:10:55.772223 23481 raft_consensus.cc:388] T 00000000000000000000000000000000 P cae51875629f4ff0ae751c8ffd16819f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 04:10:55.772249 23481 raft_consensus.cc:482] T 00000000000000000000000000000000 P cae51875629f4ff0ae751c8ffd16819f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 04:10:55.772316 23481 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cae51875629f4ff0ae751c8ffd16819f [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:10:55.773155 23481 raft_consensus.cc:504] T 00000000000000000000000000000000 P cae51875629f4ff0ae751c8ffd16819f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cae51875629f4ff0ae751c8ffd16819f" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 43951 } }
I20260810 04:10:55.773267 23481 leader_election.cc:302] T 00000000000000000000000000000000 P cae51875629f4ff0ae751c8ffd16819f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cae51875629f4ff0ae751c8ffd16819f; no voters:
I20260810 04:10:55.773473 23484 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cae51875629f4ff0ae751c8ffd16819f [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:10:55.773447 23481 leader_election.cc:288] T 00000000000000000000000000000000 P cae51875629f4ff0ae751c8ffd16819f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 04:10:55.773658 23484 raft_consensus.cc:686] T 00000000000000000000000000000000 P cae51875629f4ff0ae751c8ffd16819f [term 1 LEADER]: Becoming Leader. State: Replica: cae51875629f4ff0ae751c8ffd16819f, State: Running, Role: LEADER
I20260810 04:10:55.773849 23481 sys_catalog.cc:564] T 00000000000000000000000000000000 P cae51875629f4ff0ae751c8ffd16819f [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:10:55.773801 23484 consensus_queue.cc:237] T 00000000000000000000000000000000 P cae51875629f4ff0ae751c8ffd16819f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cae51875629f4ff0ae751c8ffd16819f" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 43951 } }
I20260810 04:10:55.774292 23485 sys_catalog.cc:455] T 00000000000000000000000000000000 P cae51875629f4ff0ae751c8ffd16819f [sys.catalog]: SysCatalogTable state changed. Reason: New leader cae51875629f4ff0ae751c8ffd16819f. Latest consensus state: current_term: 1 leader_uuid: "cae51875629f4ff0ae751c8ffd16819f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cae51875629f4ff0ae751c8ffd16819f" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 43951 } } }
I20260810 04:10:55.774303 23486 sys_catalog.cc:455] T 00000000000000000000000000000000 P cae51875629f4ff0ae751c8ffd16819f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cae51875629f4ff0ae751c8ffd16819f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cae51875629f4ff0ae751c8ffd16819f" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 43951 } } }
I20260810 04:10:55.774444 23486 sys_catalog.cc:458] T 00000000000000000000000000000000 P cae51875629f4ff0ae751c8ffd16819f [sys.catalog]: This master's current role is: LEADER
I20260810 04:10:55.774593 23485 sys_catalog.cc:458] T 00000000000000000000000000000000 P cae51875629f4ff0ae751c8ffd16819f [sys.catalog]: This master's current role is: LEADER
I20260810 04:10:55.774974 23490 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 04:10:55.775414 31872 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskws89Io/build/release/bin/kudu as pid 23404
I20260810 04:10:55.775516 23490 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 04:10:55.775552 31872 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786334794676589-31872-0/minicluster-data/master-0/wal/instance
I20260810 04:10:55.775772 31872 external_mini_cluster.cc:1428] Setting key d1c1df775fbab12768de33af25377545
I20260810 04:10:55.778283 23502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:55.776269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33939 (local address 127.31.32.62:43951)
0810 04:10:55.776449 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0810 04:10:55.776458 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 04:10:55.776758 (+ 300us) server_negotiation.cc:407] Connection header received
0810 04:10:55.776924 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:10:55.776931 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:10:55.776954 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:10:55.777086 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:10:55.777829 (+ 743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:10:55.777832 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:10:55.777832 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:10:55.777837 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:10:55.777896 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:10:55.777907 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:10:55.777978 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0810 04:10:55.778055 (+ 77us) server_negotiation.cc:299] Negotiation successful
0810 04:10:55.778100 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":104,"threads_started":1}
I20260810 04:10:55.779260 31872 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 04:10:55.784576 23490 catalog_manager.cc:1347] Generated new cluster ID: d1f741a6d2584028ae302fdd13e53dd4
I20260810 04:10:55.784628 23490 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 04:10:55.785647 23502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:55.780616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56949 (local address 127.31.32.62:43951)
0810 04:10:55.780648 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0810 04:10:55.780650 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:10:55.784633 (+ 3983us) server_negotiation.cc:407] Connection header received
0810 04:10:55.784724 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:10:55.784725 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:10:55.784731 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:10:55.784777 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:10:55.785433 (+ 656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:10:55.785434 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:10:55.785434 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:10:55.785438 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:10:55.785452 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:10:55.785456 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:10:55.785501 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0810 04:10:55.785555 (+ 54us) server_negotiation.cc:299] Negotiation successful
0810 04:10:55.785569 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260810 04:10:55.813592 23490 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 04:10:55.813828 23490 catalog_manager.cc:1504] Loading token signing keys...
I20260810 04:10:55.823559 23490 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cae51875629f4ff0ae751c8ffd16819f: Generated new TSK 0
I20260810 04:10:55.823776 23490 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 04:10:55.830971 31872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskws89Io/build/release/bin/kudu with pid 23404
2026-08-10T04:10:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1389 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786334794676589-31872-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 04:10:55 dist-test-slave-8w9v krb5kdc[23513](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 04:10:55 dist-test-slave-8w9v krb5kdc[23513](info): set up 2 sockets
Aug 10 04:10:55 dist-test-slave-8w9v krb5kdc[23513](info): commencing operation
krb5kdc: starting...
W20260810 04:10:56.921547 31872 mini_kdc.cc:121] Time spent starting KDC: real 1.070s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786334794676589-31872-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786334794676589-31872-0/krb5kdc/test-user.keytab.
Aug 10 04:10:57 dist-test-slave-8w9v krb5kdc[23513](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335057, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T04:10:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T04:10:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.32.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.32.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.32.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.62.keytab.
Entry for principal kudu/127.31.32.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.62.keytab.
I20260810 04:10:57.044620 31872 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskws89Io/build/release/bin/kudu
/tmp/dist-test-taskws89Io/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786334794676589-31872-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786334794676589-31872-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786334794676589-31872-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786334794676589-31872-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.32.62:43315
--webserver_interface=127.31.32.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.62.keytab
--principal=kudu/127.31.32.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.32.20:41509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.32.62:43315
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786334794676589-31872-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786334794676589-31872-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786334794676589-31872-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:10:57.127619 23531 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:10:57.127858 23531 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:10:57.127882 23531 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:10:57.129127 23531 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:10:57.129172 23531 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:10:57.129186 23531 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:10:57.129199 23531 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:10:57.129253 23531 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:10:57.130658 23531 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.32.20:41509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786334794676589-31872-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786334794676589-31872-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.32.62:43315
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.62.keytab
--principal=kudu/127.31.32.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.32.62:43315
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786334794676589-31872-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.32.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23531
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786334794676589-31872-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:10:57.130913 23531 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:10:57.131132 23531 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:10:57.136076 23537 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:10:57.136312 23531 server_base.cc:1034] running on GCE node
W20260810 04:10:57.136094 23538 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:10:57.136081 23541 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:10:57.136695 23531 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:10:57.137036 23531 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:10:57.138191 23531 hybrid_clock.cc:648] HybridClock initialized: now 1786335057138172 us; error 41 us; skew 500 ppm
Aug 10 04:10:57 dist-test-slave-8w9v krb5kdc[23513](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335057, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:10:57.140770 23531 init.cc:377] Logged in from keytab as kudu/127.31.32.62@KRBTEST.COM (short username kudu)
I20260810 04:10:57.141238 23531 webserver.cc:460] Webserver started at http://127.31.32.62:38575/ using document root <none> and password file <none>
I20260810 04:10:57.141475 23531 fs_manager.cc:359] Metadata directory not provided
I20260810 04:10:57.141525 23531 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:10:57.141655 23531 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:10:57.142753 23531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786334794676589-31872-0/minicluster-data/master-0/data/instance:
uuid: "f37d296cec8d49a8a4419a6d2636b2e5"
format_stamp: "Formatted at 2026-08-10 04:10:57 on dist-test-slave-8w9v"
server_key: "268e41181db0a48f319e701aaba6d605"
server_key_iv: "ede89fb6e2b56a48595e59f13cab5b59"
server_key_version: "encryptionkey@0"
I20260810 04:10:57.143127 23531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786334794676589-31872-0/minicluster-data/master-0/wal/instance:
uuid: "f37d296cec8d49a8a4419a6d2636b2e5"
format_stamp: "Formatted at 2026-08-10 04:10:57 on dist-test-slave-8w9v"
server_key: "268e41181db0a48f319e701aaba6d605"
server_key_iv: "ede89fb6e2b56a48595e59f13cab5b59"
server_key_version: "encryptionkey@0"
I20260810 04:10:57.144553 23531 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 04:10:57.145303 23549 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:10:57.145504 23531 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 04:10:57.145601 23531 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786334794676589-31872-0/minicluster-data/master-0/data,/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786334794676589-31872-0/minicluster-data/master-0/wal
uuid: "f37d296cec8d49a8a4419a6d2636b2e5"
format_stamp: "Formatted at 2026-08-10 04:10:57 on dist-test-slave-8w9v"
server_key: "268e41181db0a48f319e701aaba6d605"
server_key_iv: "ede89fb6e2b56a48595e59f13cab5b59"
server_key_version: "encryptionkey@0"
I20260810 04:10:57.145678 23531 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786334794676589-31872-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786334794676589-31872-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786334794676589-31872-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:10:57.154904 23531 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:10:57.157167 23531 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:10:57.157334 23531 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:10:57.161579 23531 rpc_server.cc:307] RPC server started. Bound to: 127.31.32.62:43315
I20260810 04:10:57.161600 23604 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.32.62:43315 every 8 connection(s)
I20260810 04:10:57.161983 23531 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786334794676589-31872-0/minicluster-data/master-0/data/info.pb
I20260810 04:10:57.162789 23605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:10:57.165159 23605 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f37d296cec8d49a8a4419a6d2636b2e5: Bootstrap starting.
I20260810 04:10:57.165750 23605 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f37d296cec8d49a8a4419a6d2636b2e5: Neither blocks nor log segments found. Creating new log.
I20260810 04:10:57.166052 23605 log.cc:824] T 00000000000000000000000000000000 P f37d296cec8d49a8a4419a6d2636b2e5: Log is configured to *not* fsync() on all Append() calls
I20260810 04:10:57.166806 23605 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f37d296cec8d49a8a4419a6d2636b2e5: No bootstrap required, opened a new log
I20260810 04:10:57.168574 23605 raft_consensus.cc:348] T 00000000000000000000000000000000 P f37d296cec8d49a8a4419a6d2636b2e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f37d296cec8d49a8a4419a6d2636b2e5" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 43315 } }
I20260810 04:10:57.168738 23605 raft_consensus.cc:374] T 00000000000000000000000000000000 P f37d296cec8d49a8a4419a6d2636b2e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:10:57.168787 23605 raft_consensus.cc:729] T 00000000000000000000000000000000 P f37d296cec8d49a8a4419a6d2636b2e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f37d296cec8d49a8a4419a6d2636b2e5, State: Initialized, Role: FOLLOWER
I20260810 04:10:57.168954 23605 consensus_queue.cc:260] T 00000000000000000000000000000000 P f37d296cec8d49a8a4419a6d2636b2e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f37d296cec8d49a8a4419a6d2636b2e5" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 43315 } }
I20260810 04:10:57.169041 23605 raft_consensus.cc:388] T 00000000000000000000000000000000 P f37d296cec8d49a8a4419a6d2636b2e5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 04:10:57.169085 23605 raft_consensus.cc:482] T 00000000000000000000000000000000 P f37d296cec8d49a8a4419a6d2636b2e5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 04:10:57.169135 23605 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f37d296cec8d49a8a4419a6d2636b2e5 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:10:57.169600 31872 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskws89Io/build/release/bin/kudu as pid 23531
I20260810 04:10:57.169721 31872 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786334794676589-31872-0/minicluster-data/master-0/wal/instance
I20260810 04:10:57.169956 31872 external_mini_cluster.cc:1428] Setting key 0ca46b32379a8ea51bb45a30818cfc2f
I20260810 04:10:57.170241 23605 raft_consensus.cc:504] T 00000000000000000000000000000000 P f37d296cec8d49a8a4419a6d2636b2e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f37d296cec8d49a8a4419a6d2636b2e5" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 43315 } }
I20260810 04:10:57.170379 23605 leader_election.cc:302] T 00000000000000000000000000000000 P f37d296cec8d49a8a4419a6d2636b2e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f37d296cec8d49a8a4419a6d2636b2e5; no voters:
I20260810 04:10:57.170557 23605 leader_election.cc:288] T 00000000000000000000000000000000 P f37d296cec8d49a8a4419a6d2636b2e5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 04:10:57.170603 23609 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f37d296cec8d49a8a4419a6d2636b2e5 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:10:57.170794 23609 raft_consensus.cc:686] T 00000000000000000000000000000000 P f37d296cec8d49a8a4419a6d2636b2e5 [term 1 LEADER]: Becoming Leader. State: Replica: f37d296cec8d49a8a4419a6d2636b2e5, State: Running, Role: LEADER
I20260810 04:10:57.170878 23605 sys_catalog.cc:564] T 00000000000000000000000000000000 P f37d296cec8d49a8a4419a6d2636b2e5 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:10:57.170955 23609 consensus_queue.cc:237] T 00000000000000000000000000000000 P f37d296cec8d49a8a4419a6d2636b2e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f37d296cec8d49a8a4419a6d2636b2e5" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 43315 } }
I20260810 04:10:57.171459 23613 sys_catalog.cc:455] T 00000000000000000000000000000000 P f37d296cec8d49a8a4419a6d2636b2e5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f37d296cec8d49a8a4419a6d2636b2e5. Latest consensus state: current_term: 1 leader_uuid: "f37d296cec8d49a8a4419a6d2636b2e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f37d296cec8d49a8a4419a6d2636b2e5" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 43315 } } }
I20260810 04:10:57.171452 23612 sys_catalog.cc:455] T 00000000000000000000000000000000 P f37d296cec8d49a8a4419a6d2636b2e5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f37d296cec8d49a8a4419a6d2636b2e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f37d296cec8d49a8a4419a6d2636b2e5" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 43315 } } }
I20260810 04:10:57.171614 23612 sys_catalog.cc:458] T 00000000000000000000000000000000 P f37d296cec8d49a8a4419a6d2636b2e5 [sys.catalog]: This master's current role is: LEADER
I20260810 04:10:57.171809 23613 sys_catalog.cc:458] T 00000000000000000000000000000000 P f37d296cec8d49a8a4419a6d2636b2e5 [sys.catalog]: This master's current role is: LEADER
I20260810 04:10:57.172230 23621 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 04:10:57.172782 23621 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 04:10:57.174425 23621 catalog_manager.cc:1347] Generated new cluster ID: bb4a352c93664bb6b97d98bfd9ca7523
I20260810 04:10:57.174480 23621 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 10 04:10:57 dist-test-slave-8w9v krb5kdc[23513](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335057, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.32.62@KRBTEST.COM
I20260810 04:10:57.181610 23610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:57.170489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43143 (local address 127.31.32.62:43315)
0810 04:10:57.170698 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0810 04:10:57.170709 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 04:10:57.171173 (+ 464us) server_negotiation.cc:407] Connection header received
0810 04:10:57.171340 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:10:57.171349 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:10:57.171414 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:10:57.171577 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:10:57.172662 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:57.173837 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:57.174783 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:57.175194 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:57.175326 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:10:57.177787 (+ 2461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:10:57.177791 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:10:57.177792 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:10:57.177804 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:10:57.180000 (+ 2196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:10:57.180399 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:10:57.180402 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:10:57.180403 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:10:57.180508 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:10:57.180723 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:10:57.180724 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:10:57.180725 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:10:57.180980 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:10:57.181149 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0810 04:10:57.181256 (+ 107us) server_negotiation.cc:299] Negotiation successful
0810 04:10:57.181408 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":100,"threads_started":1}
I20260810 04:10:57.182579 31872 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 04:10:57.188746 23621 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 04:10:57.189446 23621 catalog_manager.cc:1504] Loading token signing keys...
I20260810 04:10:57.194737 23610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:57.183996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44937 (local address 127.31.32.62:43315)
0810 04:10:57.184034 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0810 04:10:57.184036 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:10:57.188231 (+ 4195us) server_negotiation.cc:407] Connection header received
0810 04:10:57.188301 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:10:57.188302 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:10:57.188314 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:10:57.188385 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:10:57.189120 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:57.189662 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:57.190442 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:57.190657 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:57.190735 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:10:57.191997 (+ 1262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:10:57.191998 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:10:57.191998 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:10:57.192005 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:10:57.193497 (+ 1492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:10:57.193854 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:10:57.193855 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:10:57.193856 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:10:57.193917 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:10:57.194119 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:10:57.194119 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:10:57.194120 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:10:57.194237 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:10:57.194344 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0810 04:10:57.194517 (+ 173us) server_negotiation.cc:299] Negotiation successful
0810 04:10:57.194616 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260810 04:10:57.205106 23621 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f37d296cec8d49a8a4419a6d2636b2e5: Generated new TSK 0
I20260810 04:10:57.205318 23621 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 04:10:57.209414 31872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskws89Io/build/release/bin/kudu with pid 23531
2026-08-10T04:10:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1379 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786334794676589-31872-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 04:10:57 dist-test-slave-8w9v krb5kdc[23640](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 04:10:57 dist-test-slave-8w9v krb5kdc[23640](info): set up 2 sockets
Aug 10 04:10:57 dist-test-slave-8w9v krb5kdc[23640](info): commencing operation
krb5kdc: starting...
W20260810 04:10:58.322191 31872 mini_kdc.cc:121] Time spent starting KDC: real 1.091s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786334794676589-31872-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786334794676589-31872-0/krb5kdc/test-user.keytab.
Aug 10 04:10:58 dist-test-slave-8w9v krb5kdc[23640](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335058, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T04:10:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T04:10:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.32.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.32.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.32.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.62.keytab.
Entry for principal kudu/127.31.32.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.62.keytab.
I20260810 04:10:58.446777 31872 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskws89Io/build/release/bin/kudu
/tmp/dist-test-taskws89Io/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786334794676589-31872-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786334794676589-31872-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786334794676589-31872-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786334794676589-31872-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.32.62:36921
--webserver_interface=127.31.32.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.62.keytab
--principal=kudu/127.31.32.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.32.20:44685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.32.62:36921
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786334794676589-31872-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786334794676589-31872-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786334794676589-31872-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:10:58.532570 23655 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:10:58.532827 23655 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:10:58.532871 23655 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:10:58.534026 23655 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:10:58.534094 23655 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:10:58.534118 23655 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:10:58.534142 23655 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:10:58.534170 23655 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:10:58.535646 23655 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.32.20:44685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786334794676589-31872-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786334794676589-31872-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.32.62:36921
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.62.keytab
--principal=kudu/127.31.32.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.32.62:36921
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786334794676589-31872-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.32.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23655
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786334794676589-31872-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:10:58.535919 23655 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:10:58.536188 23655 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:10:58.541117 23661 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:10:58.541124 23665 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:10:58.541134 23662 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:10:58.541406 23655 server_base.cc:1034] running on GCE node
I20260810 04:10:58.541689 23655 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:10:58.542001 23655 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:10:58.543171 23655 hybrid_clock.cc:648] HybridClock initialized: now 1786335058543162 us; error 40 us; skew 500 ppm
Aug 10 04:10:58 dist-test-slave-8w9v krb5kdc[23640](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335058, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:10:58.545737 23655 init.cc:377] Logged in from keytab as kudu/127.31.32.62@KRBTEST.COM (short username kudu)
I20260810 04:10:58.546170 23655 webserver.cc:460] Webserver started at http://127.31.32.62:42521/ using document root <none> and password file <none>
I20260810 04:10:58.546428 23655 fs_manager.cc:359] Metadata directory not provided
I20260810 04:10:58.546516 23655 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:10:58.546665 23655 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:10:58.547708 23655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786334794676589-31872-0/minicluster-data/master-0/data/instance:
uuid: "88e8ef5d99a746999765bf3bf5d855c2"
format_stamp: "Formatted at 2026-08-10 04:10:58 on dist-test-slave-8w9v"
server_key: "2365b9cf192f47cca5a0a7577f0abc23"
server_key_iv: "82cf82986a433dd1398963f364453cd3"
server_key_version: "encryptionkey@0"
I20260810 04:10:58.548087 23655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786334794676589-31872-0/minicluster-data/master-0/wal/instance:
uuid: "88e8ef5d99a746999765bf3bf5d855c2"
format_stamp: "Formatted at 2026-08-10 04:10:58 on dist-test-slave-8w9v"
server_key: "2365b9cf192f47cca5a0a7577f0abc23"
server_key_iv: "82cf82986a433dd1398963f364453cd3"
server_key_version: "encryptionkey@0"
I20260810 04:10:58.549459 23655 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260810 04:10:58.550165 23673 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:10:58.550320 23655 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260810 04:10:58.550438 23655 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786334794676589-31872-0/minicluster-data/master-0/data,/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786334794676589-31872-0/minicluster-data/master-0/wal
uuid: "88e8ef5d99a746999765bf3bf5d855c2"
format_stamp: "Formatted at 2026-08-10 04:10:58 on dist-test-slave-8w9v"
server_key: "2365b9cf192f47cca5a0a7577f0abc23"
server_key_iv: "82cf82986a433dd1398963f364453cd3"
server_key_version: "encryptionkey@0"
I20260810 04:10:58.550513 23655 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786334794676589-31872-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786334794676589-31872-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786334794676589-31872-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:10:58.562243 23655 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:10:58.564062 23655 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:10:58.564283 23655 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:10:58.568311 23655 rpc_server.cc:307] RPC server started. Bound to: 127.31.32.62:36921
I20260810 04:10:58.568364 23731 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.32.62:36921 every 8 connection(s)
I20260810 04:10:58.568725 23655 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786334794676589-31872-0/minicluster-data/master-0/data/info.pb
I20260810 04:10:58.569587 23732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:10:58.571910 23732 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88e8ef5d99a746999765bf3bf5d855c2: Bootstrap starting.
I20260810 04:10:58.572594 23732 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 88e8ef5d99a746999765bf3bf5d855c2: Neither blocks nor log segments found. Creating new log.
I20260810 04:10:58.572700 31872 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskws89Io/build/release/bin/kudu as pid 23655
I20260810 04:10:58.572845 31872 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786334794676589-31872-0/minicluster-data/master-0/wal/instance
I20260810 04:10:58.572958 23732 log.cc:824] T 00000000000000000000000000000000 P 88e8ef5d99a746999765bf3bf5d855c2: Log is configured to *not* fsync() on all Append() calls
I20260810 04:10:58.573060 31872 external_mini_cluster.cc:1428] Setting key 094f93e533056de68f8a8d7d55209609
I20260810 04:10:58.573779 23732 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88e8ef5d99a746999765bf3bf5d855c2: No bootstrap required, opened a new log
I20260810 04:10:58.575891 23732 raft_consensus.cc:348] T 00000000000000000000000000000000 P 88e8ef5d99a746999765bf3bf5d855c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88e8ef5d99a746999765bf3bf5d855c2" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 36921 } }
I20260810 04:10:58.576061 23732 raft_consensus.cc:374] T 00000000000000000000000000000000 P 88e8ef5d99a746999765bf3bf5d855c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:10:58.576110 23732 raft_consensus.cc:729] T 00000000000000000000000000000000 P 88e8ef5d99a746999765bf3bf5d855c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88e8ef5d99a746999765bf3bf5d855c2, State: Initialized, Role: FOLLOWER
I20260810 04:10:58.576268 23732 consensus_queue.cc:260] T 00000000000000000000000000000000 P 88e8ef5d99a746999765bf3bf5d855c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88e8ef5d99a746999765bf3bf5d855c2" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 36921 } }
I20260810 04:10:58.576356 23732 raft_consensus.cc:388] T 00000000000000000000000000000000 P 88e8ef5d99a746999765bf3bf5d855c2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 04:10:58.576401 23732 raft_consensus.cc:482] T 00000000000000000000000000000000 P 88e8ef5d99a746999765bf3bf5d855c2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 04:10:58.576448 23732 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 88e8ef5d99a746999765bf3bf5d855c2 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:10:58.576992 23732 raft_consensus.cc:504] T 00000000000000000000000000000000 P 88e8ef5d99a746999765bf3bf5d855c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88e8ef5d99a746999765bf3bf5d855c2" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 36921 } }
I20260810 04:10:58.577133 23732 leader_election.cc:302] T 00000000000000000000000000000000 P 88e8ef5d99a746999765bf3bf5d855c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 88e8ef5d99a746999765bf3bf5d855c2; no voters:
I20260810 04:10:58.577318 23732 leader_election.cc:288] T 00000000000000000000000000000000 P 88e8ef5d99a746999765bf3bf5d855c2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 04:10:58.577382 23737 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 88e8ef5d99a746999765bf3bf5d855c2 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:10:58.577592 23732 sys_catalog.cc:564] T 00000000000000000000000000000000 P 88e8ef5d99a746999765bf3bf5d855c2 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:10:58.578117 23737 raft_consensus.cc:686] T 00000000000000000000000000000000 P 88e8ef5d99a746999765bf3bf5d855c2 [term 1 LEADER]: Becoming Leader. State: Replica: 88e8ef5d99a746999765bf3bf5d855c2, State: Running, Role: LEADER
I20260810 04:10:58.578258 23737 consensus_queue.cc:237] T 00000000000000000000000000000000 P 88e8ef5d99a746999765bf3bf5d855c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88e8ef5d99a746999765bf3bf5d855c2" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 36921 } }
I20260810 04:10:58.578732 23736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:58.573574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55507 (local address 127.31.32.62:36921)
0810 04:10:58.573737 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0810 04:10:58.573748 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 04:10:58.573783 (+ 35us) server_negotiation.cc:407] Connection header received
0810 04:10:58.574243 (+ 460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:10:58.574249 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:10:58.574303 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:10:58.574450 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:10:58.574681 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:58.575554 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:58.576509 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:58.576911 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:58.577027 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:10:58.577733 (+ 706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:10:58.577737 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:10:58.577737 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:10:58.577936 (+ 199us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:10:58.577972 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:10:58.577985 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:10:58.578077 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0810 04:10:58.578424 (+ 347us) server_negotiation.cc:299] Negotiation successful
0810 04:10:58.578491 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":76,"threads_started":1}
I20260810 04:10:58.578797 23738 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88e8ef5d99a746999765bf3bf5d855c2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "88e8ef5d99a746999765bf3bf5d855c2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88e8ef5d99a746999765bf3bf5d855c2" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 36921 } } }
I20260810 04:10:58.578890 23738 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88e8ef5d99a746999765bf3bf5d855c2 [sys.catalog]: This master's current role is: LEADER
I20260810 04:10:58.579084 23754 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 04:10:58.579097 23739 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88e8ef5d99a746999765bf3bf5d855c2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e8ef5d99a746999765bf3bf5d855c2. Latest consensus state: current_term: 1 leader_uuid: "88e8ef5d99a746999765bf3bf5d855c2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88e8ef5d99a746999765bf3bf5d855c2" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 36921 } } }
I20260810 04:10:58.579560 23739 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88e8ef5d99a746999765bf3bf5d855c2 [sys.catalog]: This master's current role is: LEADER
I20260810 04:10:58.579698 31872 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 04:10:58.580000 23754 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 04:10:58.581619 23754 catalog_manager.cc:1347] Generated new cluster ID: 6ec5d45e878c42e9bba19a7fe18a7d83
I20260810 04:10:58.581682 23754 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 04:10:58.589589 23754 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 04:10:58.590260 23754 catalog_manager.cc:1504] Loading token signing keys...
I20260810 04:10:58.591554 23736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:58.581135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52543 (local address 127.31.32.62:36921)
0810 04:10:58.581172 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0810 04:10:58.581173 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 04:10:58.581406 (+ 233us) server_negotiation.cc:407] Connection header received
0810 04:10:58.581505 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:10:58.581506 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:10:58.581516 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:10:58.581544 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:10:58.588303 (+ 6759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:58.589314 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:58.590211 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:10:58.590479 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:10:58.590551 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:10:58.591286 (+ 735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:10:58.591288 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:10:58.591288 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:10:58.591294 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:10:58.591308 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:10:58.591312 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:10:58.591378 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0810 04:10:58.591441 (+ 63us) server_negotiation.cc:299] Negotiation successful
0810 04:10:58.591477 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260810 04:10:58.602535 23754 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 88e8ef5d99a746999765bf3bf5d855c2: Generated new TSK 0
I20260810 04:10:58.602762 23754 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 04:10:58.607277 31872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskws89Io/build/release/bin/kudu with pid 23655
2026-08-10T04:10:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1397 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786334794676589-31872-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 04:10:58 dist-test-slave-8w9v krb5kdc[23767](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 04:10:58 dist-test-slave-8w9v krb5kdc[23767](info): set up 2 sockets
Aug 10 04:10:58 dist-test-slave-8w9v krb5kdc[23767](info): commencing operation
krb5kdc: starting...
W20260810 04:10:59.702054 31872 mini_kdc.cc:121] Time spent starting KDC: real 1.073s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786334794676589-31872-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786334794676589-31872-0/krb5kdc/test-user.keytab.
Aug 10 04:10:59 dist-test-slave-8w9v krb5kdc[23767](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335059, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T04:10:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T04:10:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.32.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.32.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.32.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.62.keytab.
Entry for principal kudu/127.31.32.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.62.keytab.
I20260810 04:10:59.829214 31872 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskws89Io/build/release/bin/kudu
/tmp/dist-test-taskws89Io/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786334794676589-31872-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786334794676589-31872-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786334794676589-31872-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786334794676589-31872-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.32.62:40223
--webserver_interface=127.31.32.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.62.keytab
--principal=kudu/127.31.32.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.32.20:43615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.32.62:40223
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786334794676589-31872-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786334794676589-31872-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786334794676589-31872-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:10:59.912070 23780 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:10:59.912371 23780 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:10:59.912415 23780 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:10:59.913597 23780 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:10:59.913672 23780 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:10:59.913697 23780 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:10:59.913717 23780 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:10:59.913753 23780 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:10:59.915221 23780 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.32.20:43615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786334794676589-31872-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786334794676589-31872-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.32.62:40223
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.62.keytab
--principal=kudu/127.31.32.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.32.62:40223
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786334794676589-31872-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.32.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23780
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786334794676589-31872-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:10:59.915511 23780 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:10:59.915751 23780 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:10:59.920889 23785 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:10:59.920907 23788 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:10:59.920993 23780 server_base.cc:1034] running on GCE node
W20260810 04:10:59.920902 23786 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:10:59.921392 23780 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:10:59.921689 23780 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:10:59.922864 23780 hybrid_clock.cc:648] HybridClock initialized: now 1786335059922852 us; error 47 us; skew 500 ppm
Aug 10 04:10:59 dist-test-slave-8w9v krb5kdc[23767](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335059, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:10:59.925516 23780 init.cc:377] Logged in from keytab as kudu/127.31.32.62@KRBTEST.COM (short username kudu)
I20260810 04:10:59.925988 23780 webserver.cc:460] Webserver started at http://127.31.32.62:45491/ using document root <none> and password file <none>
I20260810 04:10:59.926173 23780 fs_manager.cc:359] Metadata directory not provided
I20260810 04:10:59.926214 23780 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:10:59.926302 23780 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:10:59.927326 23780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786334794676589-31872-0/minicluster-data/master-0/data/instance:
uuid: "774e1d5a0582466097cf0943ad77bb12"
format_stamp: "Formatted at 2026-08-10 04:10:59 on dist-test-slave-8w9v"
server_key: "84735609be37636f04c166134c4a33e8"
server_key_iv: "9bc389ecbe0059635ba86f7ccfedc18d"
server_key_version: "encryptionkey@0"
I20260810 04:10:59.927639 23780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786334794676589-31872-0/minicluster-data/master-0/wal/instance:
uuid: "774e1d5a0582466097cf0943ad77bb12"
format_stamp: "Formatted at 2026-08-10 04:10:59 on dist-test-slave-8w9v"
server_key: "84735609be37636f04c166134c4a33e8"
server_key_iv: "9bc389ecbe0059635ba86f7ccfedc18d"
server_key_version: "encryptionkey@0"
I20260810 04:10:59.929109 23780 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 04:10:59.929908 23795 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:10:59.930094 23780 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260810 04:10:59.930208 23780 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786334794676589-31872-0/minicluster-data/master-0/data,/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786334794676589-31872-0/minicluster-data/master-0/wal
uuid: "774e1d5a0582466097cf0943ad77bb12"
format_stamp: "Formatted at 2026-08-10 04:10:59 on dist-test-slave-8w9v"
server_key: "84735609be37636f04c166134c4a33e8"
server_key_iv: "9bc389ecbe0059635ba86f7ccfedc18d"
server_key_version: "encryptionkey@0"
I20260810 04:10:59.930284 23780 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786334794676589-31872-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786334794676589-31872-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786334794676589-31872-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:10:59.933665 23780 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:10:59.936043 23780 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:10:59.936236 23780 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:10:59.940384 23780 rpc_server.cc:307] RPC server started. Bound to: 127.31.32.62:40223
I20260810 04:10:59.940439 23851 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.32.62:40223 every 8 connection(s)
I20260810 04:10:59.940742 23780 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786334794676589-31872-0/minicluster-data/master-0/data/info.pb
I20260810 04:10:59.941633 23853 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:10:59.943908 23853 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 774e1d5a0582466097cf0943ad77bb12: Bootstrap starting.
I20260810 04:10:59.944319 31872 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskws89Io/build/release/bin/kudu as pid 23780
I20260810 04:10:59.944458 31872 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786334794676589-31872-0/minicluster-data/master-0/wal/instance
I20260810 04:10:59.944661 31872 external_mini_cluster.cc:1428] Setting key ae597c23941d49452eeb4c39666019c2
I20260810 04:10:59.944708 23853 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 774e1d5a0582466097cf0943ad77bb12: Neither blocks nor log segments found. Creating new log.
I20260810 04:10:59.945001 23853 log.cc:824] T 00000000000000000000000000000000 P 774e1d5a0582466097cf0943ad77bb12: Log is configured to *not* fsync() on all Append() calls
I20260810 04:10:59.945781 23853 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 774e1d5a0582466097cf0943ad77bb12: No bootstrap required, opened a new log
I20260810 04:10:59.947553 23858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:59.945138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56141 (local address 127.31.32.62:40223)
0810 04:10:59.945460 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0810 04:10:59.945470 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 04:10:59.945504 (+ 34us) server_negotiation.cc:407] Connection header received
0810 04:10:59.946161 (+ 657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:10:59.946170 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:10:59.946204 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:10:59.946394 (+ 190us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:10:59.947036 (+ 642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:10:59.947040 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:10:59.947040 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:10:59.947046 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:10:59.947091 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:10:59.947107 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:10:59.947200 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0810 04:10:59.947313 (+ 113us) server_negotiation.cc:299] Negotiation successful
0810 04:10:59.947363 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":88,"threads_started":1}
I20260810 04:10:59.948396 23853 raft_consensus.cc:348] T 00000000000000000000000000000000 P 774e1d5a0582466097cf0943ad77bb12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "774e1d5a0582466097cf0943ad77bb12" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 40223 } }
I20260810 04:10:59.948563 31872 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 04:10:59.948557 23853 raft_consensus.cc:374] T 00000000000000000000000000000000 P 774e1d5a0582466097cf0943ad77bb12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:10:59.948606 23853 raft_consensus.cc:729] T 00000000000000000000000000000000 P 774e1d5a0582466097cf0943ad77bb12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 774e1d5a0582466097cf0943ad77bb12, State: Initialized, Role: FOLLOWER
I20260810 04:10:59.948725 23853 consensus_queue.cc:260] T 00000000000000000000000000000000 P 774e1d5a0582466097cf0943ad77bb12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "774e1d5a0582466097cf0943ad77bb12" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 40223 } }
I20260810 04:10:59.948809 23853 raft_consensus.cc:388] T 00000000000000000000000000000000 P 774e1d5a0582466097cf0943ad77bb12 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 04:10:59.948845 23853 raft_consensus.cc:482] T 00000000000000000000000000000000 P 774e1d5a0582466097cf0943ad77bb12 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 04:10:59.948881 23853 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 774e1d5a0582466097cf0943ad77bb12 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:10:59.949611 23853 raft_consensus.cc:504] T 00000000000000000000000000000000 P 774e1d5a0582466097cf0943ad77bb12 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "774e1d5a0582466097cf0943ad77bb12" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 40223 } }
I20260810 04:10:59.949748 23853 leader_election.cc:302] T 00000000000000000000000000000000 P 774e1d5a0582466097cf0943ad77bb12 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 774e1d5a0582466097cf0943ad77bb12; no voters:
I20260810 04:10:59.949932 23853 leader_election.cc:288] T 00000000000000000000000000000000 P 774e1d5a0582466097cf0943ad77bb12 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 04:10:59.949981 23863 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 774e1d5a0582466097cf0943ad77bb12 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:10:59.950152 23863 raft_consensus.cc:686] T 00000000000000000000000000000000 P 774e1d5a0582466097cf0943ad77bb12 [term 1 LEADER]: Becoming Leader. State: Replica: 774e1d5a0582466097cf0943ad77bb12, State: Running, Role: LEADER
I20260810 04:10:59.950177 23853 sys_catalog.cc:564] T 00000000000000000000000000000000 P 774e1d5a0582466097cf0943ad77bb12 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:10:59.950269 23863 consensus_queue.cc:237] T 00000000000000000000000000000000 P 774e1d5a0582466097cf0943ad77bb12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "774e1d5a0582466097cf0943ad77bb12" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 40223 } }
I20260810 04:10:59.950587 23863 sys_catalog.cc:455] T 00000000000000000000000000000000 P 774e1d5a0582466097cf0943ad77bb12 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 774e1d5a0582466097cf0943ad77bb12. Latest consensus state: current_term: 1 leader_uuid: "774e1d5a0582466097cf0943ad77bb12" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "774e1d5a0582466097cf0943ad77bb12" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 40223 } } }
I20260810 04:10:59.950682 23863 sys_catalog.cc:458] T 00000000000000000000000000000000 P 774e1d5a0582466097cf0943ad77bb12 [sys.catalog]: This master's current role is: LEADER
I20260810 04:10:59.950598 23864 sys_catalog.cc:455] T 00000000000000000000000000000000 P 774e1d5a0582466097cf0943ad77bb12 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "774e1d5a0582466097cf0943ad77bb12" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "774e1d5a0582466097cf0943ad77bb12" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 40223 } } }
I20260810 04:10:59.950839 23864 sys_catalog.cc:458] T 00000000000000000000000000000000 P 774e1d5a0582466097cf0943ad77bb12 [sys.catalog]: This master's current role is: LEADER
I20260810 04:10:59.950935 23871 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 04:10:59.951503 23858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:10:59.950195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51433 (local address 127.31.32.62:40223)
0810 04:10:59.950406 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0810 04:10:59.950408 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:10:59.950418 (+ 10us) server_negotiation.cc:407] Connection header received
0810 04:10:59.950440 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:10:59.950441 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:10:59.950445 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:10:59.950517 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:10:59.951113 (+ 596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:10:59.951115 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:10:59.951115 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:10:59.951119 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:10:59.951134 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:10:59.951138 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:10:59.951410 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0810 04:10:59.951431 (+ 21us) server_negotiation.cc:299] Negotiation successful
0810 04:10:59.951443 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":193}
I20260810 04:10:59.951707 23871 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 04:10:59.953202 23871 catalog_manager.cc:1347] Generated new cluster ID: 70859be90377409baf7233d6523a9fc6
I20260810 04:10:59.953254 23871 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 04:10:59.966297 23871 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 04:10:59.966482 23871 catalog_manager.cc:1504] Loading token signing keys...
I20260810 04:10:59.973438 23871 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 774e1d5a0582466097cf0943ad77bb12: Generated new TSK 0
I20260810 04:10:59.973654 23871 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 04:10:59.981571 31872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskws89Io/build/release/bin/kudu with pid 23780
2026-08-10T04:10:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1374 ms)
[----------] 18 tests from AuthTokenIssuingTest (25029 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786334794676589-31872-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 04:11:00 dist-test-slave-8w9v krb5kdc[23885](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 04:11:00 dist-test-slave-8w9v krb5kdc[23885](info): set up 2 sockets
Aug 10 04:11:00 dist-test-slave-8w9v krb5kdc[23885](info): commencing operation
krb5kdc: starting...
W20260810 04:11:01.077029 31872 mini_kdc.cc:121] Time spent starting KDC: real 1.073s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786334794676589-31872-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786334794676589-31872-0/krb5kdc/test-user.keytab.
Aug 10 04:11:01 dist-test-slave-8w9v krb5kdc[23885](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335061, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T04:11:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T04:11:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.32.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.32.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.32.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.62.keytab.
Entry for principal kudu/127.31.32.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.62.keytab.
I20260810 04:11:01.201411 31872 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskws89Io/build/release/bin/kudu
/tmp/dist-test-taskws89Io/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786334794676589-31872-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786334794676589-31872-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786334794676589-31872-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786334794676589-31872-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.32.62:35493
--webserver_interface=127.31.32.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.62.keytab
--principal=kudu/127.31.32.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.32.20:42711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.32.62:35493,127.31.32.61:45243,127.31.32.60:37165
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786334794676589-31872-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786334794676589-31872-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786334794676589-31872-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:11:01.283747 23900 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 04:11:01.284011 23900 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:11:01.284044 23900 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:11:01.284089 23900 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:11:01.285220 23900 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:11:01.285296 23900 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:11:01.285331 23900 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:11:01.285364 23900 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:11:01.285386 23900 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:11:01.286881 23900 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.32.20:42711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786334794676589-31872-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786334794676589-31872-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.32.62:35493,127.31.32.61:45243,127.31.32.60:37165
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.62.keytab
--principal=kudu/127.31.32.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.32.62:35493
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786334794676589-31872-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.32.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23900
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786334794676589-31872-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:11:01.287161 23900 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:11:01.287417 23900 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:11:01.292402 23907 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:11:01.292477 23900 server_base.cc:1034] running on GCE node
W20260810 04:11:01.292377 23906 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:11:01.292557 23910 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:11:01.292802 23900 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:11:01.293130 23900 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:11:01.294318 23900 hybrid_clock.cc:648] HybridClock initialized: now 1786335061294313 us; error 39 us; skew 500 ppm
Aug 10 04:11:01 dist-test-slave-8w9v krb5kdc[23885](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335061, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:11:01.302003 23900 init.cc:377] Logged in from keytab as kudu/127.31.32.62@KRBTEST.COM (short username kudu)
I20260810 04:11:01.302513 23900 webserver.cc:460] Webserver started at http://127.31.32.62:43735/ using document root <none> and password file <none>
I20260810 04:11:01.302755 23900 fs_manager.cc:359] Metadata directory not provided
I20260810 04:11:01.302835 23900 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:11:01.302992 23900 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:11:01.304042 23900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786334794676589-31872-0/minicluster-data/master-0/data/instance:
uuid: "14543a50c1dd47ee994208d8f3c5d27f"
format_stamp: "Formatted at 2026-08-10 04:11:01 on dist-test-slave-8w9v"
server_key: "e32ec0a1807e8a336632450528c46b9a"
server_key_iv: "7fa680584d8b46d48a9582ef3a9b2480"
server_key_version: "encryptionkey@0"
I20260810 04:11:01.304474 23900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786334794676589-31872-0/minicluster-data/master-0/wal/instance:
uuid: "14543a50c1dd47ee994208d8f3c5d27f"
format_stamp: "Formatted at 2026-08-10 04:11:01 on dist-test-slave-8w9v"
server_key: "e32ec0a1807e8a336632450528c46b9a"
server_key_iv: "7fa680584d8b46d48a9582ef3a9b2480"
server_key_version: "encryptionkey@0"
I20260810 04:11:01.305866 23900 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:11:01.306742 23918 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:11:01.306932 23900 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:11:01.307039 23900 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786334794676589-31872-0/minicluster-data/master-0/data,/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786334794676589-31872-0/minicluster-data/master-0/wal
uuid: "14543a50c1dd47ee994208d8f3c5d27f"
format_stamp: "Formatted at 2026-08-10 04:11:01 on dist-test-slave-8w9v"
server_key: "e32ec0a1807e8a336632450528c46b9a"
server_key_iv: "7fa680584d8b46d48a9582ef3a9b2480"
server_key_version: "encryptionkey@0"
I20260810 04:11:01.307114 23900 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786334794676589-31872-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786334794676589-31872-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786334794676589-31872-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:11:01.317562 23900 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:11:01.319659 23900 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:11:01.319818 23900 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:11:01.324090 23900 rpc_server.cc:307] RPC server started. Bound to: 127.31.32.62:35493
I20260810 04:11:01.324150 23972 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.32.62:35493 every 8 connection(s)
I20260810 04:11:01.324563 23900 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786334794676589-31872-0/minicluster-data/master-0/data/info.pb
I20260810 04:11:01.325500 23973 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:11:01.326882 31872 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskws89Io/build/release/bin/kudu as pid 23900
I20260810 04:11:01.326985 31872 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786334794676589-31872-0/minicluster-data/master-0/wal/instance
I20260810 04:11:01.327159 31872 external_mini_cluster.cc:1428] Setting key c904ea8baa54a0194c186f2f02ee41b0
I20260810 04:11:01.326967 23973 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.32.62" port: 35493 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 04:11:01 dist-test-slave-8w9v krb5kdc[23885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335061, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.62@KRBTEST.COM for kudu/127.31.32.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.31.32.61@KRBTEST.COM; defaulting to no policy
I20260810 04:11:01.337541 23975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:11:01.327710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:35493 (local address 127.0.0.1:38266)
0810 04:11:01.327905 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0810 04:11:01.327948 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 04:11:01.328525 (+ 577us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:11:01.328841 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:11:01.328843 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:11:01.329111 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:11:01.329255 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:11:01.329259 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:11:01.330389 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:11:01.330390 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:11:01.331052 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:11:01.331053 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:11:01.331633 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:11:01.331635 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:11:01.331699 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:11:01.332317 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:11:01.332326 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:11:01.334013 (+ 1687us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:11:01.335879 (+ 1866us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:11:01.335886 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:11:01.335888 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:11:01.336095 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:11:01.336466 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:11:01.336466 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:11:01.336467 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:11:01.336553 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:11:01.336921 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:11:01.336926 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:11:01.336988 (+ 62us) client_negotiation.cc:769] Sending connection context
0810 04:11:01.337087 (+ 99us) client_negotiation.cc:240] Negotiation successful
0810 04:11:01.337296 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"mutex_wait_us":3,"thread_start_us":105,"threads_started":1}
I20260810 04:11:01.337544 23976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:11:01.327744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38266 (local address 127.31.32.62:35493)
0810 04:11:01.327948 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0810 04:11:01.327957 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 04:11:01.328092 (+ 135us) server_negotiation.cc:407] Connection header received
0810 04:11:01.328690 (+ 598us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:11:01.328692 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:11:01.328712 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:11:01.328833 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:11:01.329406 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:11:01.330285 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:11:01.331205 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:11:01.331516 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:11:01.331656 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:11:01.334124 (+ 2468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:11:01.334132 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:11:01.334132 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:11:01.334139 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:11:01.335790 (+ 1651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:11:01.336298 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:11:01.336303 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:11:01.336303 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:11:01.336390 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:11:01.336631 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:11:01.336631 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:11:01.336632 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:11:01.336841 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:11:01.336929 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0810 04:11:01.337137 (+ 208us) server_negotiation.cc:299] Negotiation successful
0810 04:11:01.337349 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":127,"server-negotiator.queue_time_us":176,"thread_start_us":111,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.32.61@KRBTEST.COM" created.
I20260810 04:11:01.338460 23973 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.32.61" port: 45243 } has no permanent_uuid. Determining permanent_uuid...
I20260810 04:11:01.338850 23975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:11:01.338695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.61:45243 (local address 127.0.0.1:56468)
0810 04:11:01.338750 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0810 04:11:01.338813 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.61:45243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260810 04:11:01.339016 23923 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.32.61:45243: connect: Connection refused (error 111)
I20260810 04:11:01.339525 23975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:11:01.339401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.61:45243 (local address 127.0.0.1:56482)
0810 04:11:01.339461 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0810 04:11:01.339493 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.61:45243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260810 04:11:01.339636 23973 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.32.61:45243: Network error: Client connection negotiation failed: client connection to 127.31.32.61:45243: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.32.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.61.keytab.
Entry for principal kudu/127.31.32.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.61.keytab.
I20260810 04:11:01.353368 31872 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskws89Io/build/release/bin/kudu
/tmp/dist-test-taskws89Io/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786334794676589-31872-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786334794676589-31872-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786334794676589-31872-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786334794676589-31872-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.32.61:45243
--webserver_interface=127.31.32.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.61.keytab
--principal=kudu/127.31.32.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.32.20:42711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.32.62:35493,127.31.32.61:45243,127.31.32.60:37165
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786334794676589-31872-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786334794676589-31872-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786334794676589-31872-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 04:11:01.388890 23973 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.32.61" port: 45243 } attempt: 1
I20260810 04:11:01.389462 23975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:11:01.389293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.61:45243 (local address 127.0.0.1:56492)
0810 04:11:01.389365 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0810 04:11:01.389424 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.61:45243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260810 04:11:01.389884 23975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:11:01.389764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.61:45243 (local address 127.0.0.1:56494)
0810 04:11:01.389817 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0810 04:11:01.389852 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.61:45243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260810 04:11:01.390027 23973 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.32.61:45243: Network error: Client connection negotiation failed: client connection to 127.31.32.61:45243: connect: Connection refused (error 111)
W20260810 04:11:01.435621 23982 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 04:11:01.435823 23982 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:11:01.435840 23982 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:11:01.435859 23982 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:11:01.437180 23982 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:11:01.437230 23982 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:11:01.437245 23982 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:11:01.437258 23982 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:11:01.437270 23982 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:11:01.438786 23982 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.32.20:42711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786334794676589-31872-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786334794676589-31872-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.31.32.62:35493,127.31.32.61:45243,127.31.32.60:37165
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.61.keytab
--principal=kudu/127.31.32.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.32.61:45243
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786334794676589-31872-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.31.32.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23982
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786334794676589-31872-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:11:01.439046 23982 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:11:01.439271 23982 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:11:01.444197 23989 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:11:01.444221 23988 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:11:01.444232 23982 server_base.cc:1034] running on GCE node
W20260810 04:11:01.444204 23991 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:11:01.444691 23982 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:11:01.444989 23982 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:11:01.446156 23982 hybrid_clock.cc:648] HybridClock initialized: now 1786335061446147 us; error 43 us; skew 500 ppm
Aug 10 04:11:01 dist-test-slave-8w9v krb5kdc[23885](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335061, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:11:01.448859 23982 init.cc:377] Logged in from keytab as kudu/127.31.32.61@KRBTEST.COM (short username kudu)
I20260810 04:11:01.449288 23982 webserver.cc:460] Webserver started at http://127.31.32.61:35005/ using document root <none> and password file <none>
I20260810 04:11:01.449536 23982 fs_manager.cc:359] Metadata directory not provided
I20260810 04:11:01.449605 23982 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:11:01.449759 23982 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:11:01.450865 23982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786334794676589-31872-0/minicluster-data/master-1/data/instance:
uuid: "40a0dbb48ab54d009a6e59de7a258bd6"
format_stamp: "Formatted at 2026-08-10 04:11:01 on dist-test-slave-8w9v"
server_key: "fc5bb955a49cc0948260de85a3a6f956"
server_key_iv: "1dc2fc5036470dc2945596fe6f935483"
server_key_version: "encryptionkey@0"
I20260810 04:11:01.451225 23982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786334794676589-31872-0/minicluster-data/master-1/wal/instance:
uuid: "40a0dbb48ab54d009a6e59de7a258bd6"
format_stamp: "Formatted at 2026-08-10 04:11:01 on dist-test-slave-8w9v"
server_key: "fc5bb955a49cc0948260de85a3a6f956"
server_key_iv: "1dc2fc5036470dc2945596fe6f935483"
server_key_version: "encryptionkey@0"
I20260810 04:11:01.452653 23982 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 04:11:01.453377 23999 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:11:01.453539 23982 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 04:11:01.453644 23982 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786334794676589-31872-0/minicluster-data/master-1/data,/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786334794676589-31872-0/minicluster-data/master-1/wal
uuid: "40a0dbb48ab54d009a6e59de7a258bd6"
format_stamp: "Formatted at 2026-08-10 04:11:01 on dist-test-slave-8w9v"
server_key: "fc5bb955a49cc0948260de85a3a6f956"
server_key_iv: "1dc2fc5036470dc2945596fe6f935483"
server_key_version: "encryptionkey@0"
I20260810 04:11:01.453716 23982 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786334794676589-31872-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786334794676589-31872-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786334794676589-31872-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:11:01.458194 23973 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.32.61" port: 45243 } attempt: 2
I20260810 04:11:01.458630 23975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:11:01.458503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.61:45243 (local address 127.0.0.1:56506)
0810 04:11:01.458562 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0810 04:11:01.458600 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.61:45243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260810 04:11:01.458971 23975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:11:01.458875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.61:45243 (local address 127.0.0.1:56510)
0810 04:11:01.458919 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0810 04:11:01.458943 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.61:45243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260810 04:11:01.459059 23973 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.32.61:45243: Network error: Client connection negotiation failed: client connection to 127.31.32.61:45243: connect: Connection refused (error 111)
I20260810 04:11:01.465502 23982 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:11:01.467792 23982 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:11:01.467954 23982 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:11:01.471951 23982 rpc_server.cc:307] RPC server started. Bound to: 127.31.32.61:45243
I20260810 04:11:01.471997 24064 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.32.61:45243 every 8 connection(s)
I20260810 04:11:01.472383 23982 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786334794676589-31872-0/minicluster-data/master-1/data/info.pb
I20260810 04:11:01.473356 24065 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:11:01.474843 24065 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.32.62" port: 35493 } has no permanent_uuid. Determining permanent_uuid...
I20260810 04:11:01.478577 31872 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskws89Io/build/release/bin/kudu as pid 23982
I20260810 04:11:01.478691 31872 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786334794676589-31872-0/minicluster-data/master-1/wal/instance
I20260810 04:11:01.478888 31872 external_mini_cluster.cc:1428] Setting key d671937f8eb6eabea84af4af898cd37c
Aug 10 04:11:01 dist-test-slave-8w9v krb5kdc[23885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335061, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.61@KRBTEST.COM for kudu/127.31.32.62@KRBTEST.COM
I20260810 04:11:01.484496 23976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:11:01.475546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38280 (local address 127.31.32.62:35493)
0810 04:11:01.475613 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0810 04:11:01.475617 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 04:11:01.475834 (+ 217us) server_negotiation.cc:407] Connection header received
0810 04:11:01.476427 (+ 593us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:11:01.476429 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:11:01.476441 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:11:01.476498 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:11:01.477093 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:11:01.477590 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:11:01.478546 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:11:01.478811 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:11:01.478893 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:11:01.481348 (+ 2455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:11:01.481350 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:11:01.481350 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:11:01.481359 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:11:01.482839 (+ 1480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:11:01.483405 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:11:01.483406 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:11:01.483407 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:11:01.483471 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:11:01.483765 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:11:01.483766 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:11:01.483766 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:11:01.483871 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:11:01.483934 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0810 04:11:01.484338 (+ 404us) server_negotiation.cc:299] Negotiation successful
0810 04:11:01.484429 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260810 04:11:01.484771 24067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:11:01.475535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:35493 (local address 127.0.0.1:38280)
0810 04:11:01.475712 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0810 04:11:01.475748 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0810 04:11:01.476313 (+ 565us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:11:01.476569 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:11:01.476572 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:11:01.476805 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:11:01.476996 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:11:01.477000 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:11:01.477688 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:11:01.477689 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:11:01.478455 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:11:01.478457 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:11:01.478914 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:11:01.478915 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:11:01.479090 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:11:01.479638 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:11:01.479647 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:11:01.481250 (+ 1603us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:11:01.482932 (+ 1682us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:11:01.482939 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:11:01.482941 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:11:01.483309 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:11:01.483544 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:11:01.483545 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:11:01.483545 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:11:01.483694 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:11:01.483938 (+ 244us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:11:01.483946 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:11:01.484144 (+ 198us) client_negotiation.cc:769] Sending connection context
0810 04:11:01.484287 (+ 143us) client_negotiation.cc:240] Negotiation successful
0810 04:11:01.484479 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":84,"threads_started":1}
I20260810 04:11:01.485369 24065 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.32.61" port: 45243 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.31.32.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.32.60@KRBTEST.COM" created.
Aug 10 04:11:01 dist-test-slave-8w9v krb5kdc[23885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335061, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.61@KRBTEST.COM for kudu/127.31.32.61@KRBTEST.COM
I20260810 04:11:01.495147 24067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:11:01.485661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.61:45243 (local address 127.0.0.1:56516)
0810 04:11:01.485718 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0810 04:11:01.485731 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 04:11:01.485776 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:11:01.486024 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:11:01.486025 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:11:01.486177 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:11:01.486237 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:11:01.486239 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:11:01.487090 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:11:01.487092 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:11:01.487939 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:11:01.487942 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:11:01.488480 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:11:01.488481 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:11:01.488541 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:11:01.489025 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:11:01.489030 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:11:01.491720 (+ 2690us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:11:01.493990 (+ 2270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:11:01.493992 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:11:01.493992 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:11:01.494162 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:11:01.494406 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:11:01.494407 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:11:01.494407 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:11:01.494467 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:11:01.494775 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:11:01.494776 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:11:01.494816 (+ 40us) client_negotiation.cc:769] Sending connection context
0810 04:11:01.494898 (+ 82us) client_negotiation.cc:240] Negotiation successful
0810 04:11:01.495011 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260810 04:11:01.495174 24069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:11:01.485718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56516 (local address 127.31.32.61:45243)
0810 04:11:01.485877 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0810 04:11:01.485884 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 04:11:01.485905 (+ 21us) server_negotiation.cc:407] Connection header received
0810 04:11:01.485924 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:11:01.485926 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:11:01.485937 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:11:01.485999 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:11:01.486326 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:11:01.486995 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:11:01.488022 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:11:01.488397 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:11:01.488455 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:11:01.491813 (+ 3358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:11:01.491823 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:11:01.491823 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:11:01.491830 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:11:01.493895 (+ 2065us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:11:01.494242 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:11:01.494245 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:11:01.494245 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:11:01.494301 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:11:01.494540 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:11:01.494541 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:11:01.494542 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:11:01.494695 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:11:01.494786 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0810 04:11:01.494929 (+ 143us) server_negotiation.cc:299] Negotiation successful
0810 04:11:01.495065 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":69,"server-negotiator.queue_time_us":131,"thread_start_us":102,"threads_started":1}
I20260810 04:11:01.495918 24065 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.32.60" port: 37165 } has no permanent_uuid. Determining permanent_uuid...
I20260810 04:11:01.496340 24067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:11:01.496232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.60:37165 (local address 127.0.0.1:58740)
0810 04:11:01.496274 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0810 04:11:01.496316 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.60:37165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260810 04:11:01.496512 24006 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.32.60:37165: connect: Connection refused (error 111)
I20260810 04:11:01.496955 24067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:11:01.496864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.60:37165 (local address 127.0.0.1:58748)
0810 04:11:01.496914 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0810 04:11:01.496938 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.60:37165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260810 04:11:01.497082 24065 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.32.60:37165: Network error: Client connection negotiation failed: client connection to 127.31.32.60:37165: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.32.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.60.keytab.
Entry for principal kudu/127.31.32.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.60.keytab.
I20260810 04:11:01.507375 31872 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskws89Io/build/release/bin/kudu
/tmp/dist-test-taskws89Io/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786334794676589-31872-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786334794676589-31872-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786334794676589-31872-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786334794676589-31872-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.32.60:37165
--webserver_interface=127.31.32.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.60.keytab
--principal=kudu/127.31.32.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.32.20:42711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.32.62:35493,127.31.32.61:45243,127.31.32.60:37165
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786334794676589-31872-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786334794676589-31872-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786334794676589-31872-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 04:11:01.546341 24065 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.32.60" port: 37165 } attempt: 1
I20260810 04:11:01.546904 24067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:11:01.546709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.60:37165 (local address 127.0.0.1:58752)
0810 04:11:01.546781 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0810 04:11:01.546859 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.60:37165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260810 04:11:01.547331 24067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:11:01.547215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.60:37165 (local address 127.0.0.1:58766)
0810 04:11:01.547279 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0810 04:11:01.547309 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.60:37165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260810 04:11:01.547468 24065 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.32.60:37165: Network error: Client connection negotiation failed: client connection to 127.31.32.60:37165: connect: Connection refused (error 111)
I20260810 04:11:01.550211 23973 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.32.61" port: 45243 } attempt: 3
Aug 10 04:11:01 dist-test-slave-8w9v krb5kdc[23885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335061, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.62@KRBTEST.COM for kudu/127.31.32.61@KRBTEST.COM
I20260810 04:11:01.557508 23975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:11:01.550488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.61:45243 (local address 127.0.0.1:56526)
0810 04:11:01.550550 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0810 04:11:01.550562 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 04:11:01.550620 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:11:01.550784 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:11:01.550785 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:11:01.550945 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:11:01.551011 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:11:01.551012 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:11:01.551963 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:11:01.551964 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:11:01.552553 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:11:01.552555 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:11:01.553054 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:11:01.553055 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:11:01.553113 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:11:01.553600 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:11:01.553605 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:11:01.554782 (+ 1177us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:11:01.556529 (+ 1747us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:11:01.556530 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:11:01.556530 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:11:01.556713 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:11:01.556901 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:11:01.556901 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:11:01.556901 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:11:01.556954 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:11:01.557208 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:11:01.557209 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:11:01.557272 (+ 63us) client_negotiation.cc:769] Sending connection context
0810 04:11:01.557340 (+ 68us) client_negotiation.cc:240] Negotiation successful
0810 04:11:01.557419 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260810 04:11:01.557550 24069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:11:01.550569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56526 (local address 127.31.32.61:45243)
0810 04:11:01.550612 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0810 04:11:01.550614 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:11:01.550626 (+ 12us) server_negotiation.cc:407] Connection header received
0810 04:11:01.550695 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:11:01.550696 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:11:01.550709 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:11:01.550765 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:11:01.551085 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:11:01.551875 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:11:01.552629 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:11:01.552975 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:11:01.553050 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:11:01.554870 (+ 1820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:11:01.554872 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:11:01.554872 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:11:01.554880 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:11:01.556439 (+ 1559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:11:01.556769 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:11:01.556770 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:11:01.556771 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:11:01.556839 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:11:01.557014 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:11:01.557015 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:11:01.557015 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:11:01.557144 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:11:01.557211 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0810 04:11:01.557384 (+ 173us) server_negotiation.cc:299] Negotiation successful
0810 04:11:01.557463 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260810 04:11:01.557971 23973 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.32.60" port: 37165 } has no permanent_uuid. Determining permanent_uuid...
I20260810 04:11:01.558317 23975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:11:01.558212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.60:37165 (local address 127.0.0.1:58770)
0810 04:11:01.558259 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0810 04:11:01.558286 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.60:37165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260810 04:11:01.558619 23975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:11:01.558562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.60:37165 (local address 127.0.0.1:58778)
0810 04:11:01.558584 (+ 22us) negotiation.cc:107] Waiting for socket to connect
0810 04:11:01.558606 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.60:37165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14}
W20260810 04:11:01.558724 23973 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.32.60:37165: Network error: Client connection negotiation failed: client connection to 127.31.32.60:37165: connect: Connection refused (error 111)
I20260810 04:11:01.589913 23973 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.32.60" port: 37165 } attempt: 1
I20260810 04:11:01.590472 23975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:11:01.590299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.60:37165 (local address 127.0.0.1:58788)
0810 04:11:01.590382 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0810 04:11:01.590433 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.60:37165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260810 04:11:01.590883 23975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:11:01.590776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.60:37165 (local address 127.0.0.1:58800)
0810 04:11:01.590827 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0810 04:11:01.590858 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.60:37165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260810 04:11:01.590986 23973 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.32.60:37165: Network error: Client connection negotiation failed: client connection to 127.31.32.60:37165: connect: Connection refused (error 111)
W20260810 04:11:01.593823 24073 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 04:11:01.594012 24073 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:11:01.594028 24073 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:11:01.594096 24073 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:11:01.595260 24073 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:11:01.595305 24073 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:11:01.595319 24073 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:11:01.595356 24073 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:11:01.595382 24073 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:11:01.596935 24073 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.32.20:42711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786334794676589-31872-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786334794676589-31872-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.31.32.62:35493,127.31.32.61:45243,127.31.32.60:37165
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.60.keytab
--principal=kudu/127.31.32.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.32.60:37165
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786334794676589-31872-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.31.32.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24073
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786334794676589-31872-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:11:01.597198 24073 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:11:01.597440 24073 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:11:01.602315 24078 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:11:01.602415 24073 server_base.cc:1034] running on GCE node
W20260810 04:11:01.602373 24079 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:11:01.602622 24081 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:11:01.602932 24073 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:11:01.603190 24073 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:11:01.604363 24073 hybrid_clock.cc:648] HybridClock initialized: now 1786335061604347 us; error 34 us; skew 500 ppm
Aug 10 04:11:01 dist-test-slave-8w9v krb5kdc[23885](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335061, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:11:01.606947 24073 init.cc:377] Logged in from keytab as kudu/127.31.32.60@KRBTEST.COM (short username kudu)
I20260810 04:11:01.607331 24073 webserver.cc:460] Webserver started at http://127.31.32.60:37595/ using document root <none> and password file <none>
I20260810 04:11:01.607532 24073 fs_manager.cc:359] Metadata directory not provided
I20260810 04:11:01.607573 24073 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:11:01.607663 24073 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:11:01.608793 24073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786334794676589-31872-0/minicluster-data/master-2/data/instance:
uuid: "0c3ee222ef2949c1ae46261b3f9114af"
format_stamp: "Formatted at 2026-08-10 04:11:01 on dist-test-slave-8w9v"
server_key: "913368c0358ebb31926f0a699e41eadf"
server_key_iv: "313a8a98ae83fd98af5259626ff3005e"
server_key_version: "encryptionkey@0"
I20260810 04:11:01.609215 24073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786334794676589-31872-0/minicluster-data/master-2/wal/instance:
uuid: "0c3ee222ef2949c1ae46261b3f9114af"
format_stamp: "Formatted at 2026-08-10 04:11:01 on dist-test-slave-8w9v"
server_key: "913368c0358ebb31926f0a699e41eadf"
server_key_iv: "313a8a98ae83fd98af5259626ff3005e"
server_key_version: "encryptionkey@0"
I20260810 04:11:01.610623 24073 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 04:11:01.611335 24088 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:11:01.611526 24073 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:11:01.611598 24073 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786334794676589-31872-0/minicluster-data/master-2/data,/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786334794676589-31872-0/minicluster-data/master-2/wal
uuid: "0c3ee222ef2949c1ae46261b3f9114af"
format_stamp: "Formatted at 2026-08-10 04:11:01 on dist-test-slave-8w9v"
server_key: "913368c0358ebb31926f0a699e41eadf"
server_key_iv: "313a8a98ae83fd98af5259626ff3005e"
server_key_version: "encryptionkey@0"
I20260810 04:11:01.611649 24073 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786334794676589-31872-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786334794676589-31872-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786334794676589-31872-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:11:01.615624 24065 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.32.60" port: 37165 } attempt: 2
I20260810 04:11:01.616009 24067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:11:01.615875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.60:37165 (local address 127.0.0.1:58816)
0810 04:11:01.615932 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0810 04:11:01.615967 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.60:37165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260810 04:11:01.616403 24067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:11:01.616318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.60:37165 (local address 127.0.0.1:58824)
0810 04:11:01.616359 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0810 04:11:01.616385 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.60:37165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260810 04:11:01.616506 24065 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.32.60:37165: Network error: Client connection negotiation failed: client connection to 127.31.32.60:37165: connect: Connection refused (error 111)
I20260810 04:11:01.645064 24073 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:11:01.647032 24073 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:11:01.647199 24073 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:11:01.651470 24073 rpc_server.cc:307] RPC server started. Bound to: 127.31.32.60:37165
I20260810 04:11:01.651532 24146 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.32.60:37165 every 8 connection(s)
I20260810 04:11:01.651885 24073 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786334794676589-31872-0/minicluster-data/master-2/data/info.pb
I20260810 04:11:01.652949 31872 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskws89Io/build/release/bin/kudu as pid 24073
I20260810 04:11:01.652958 24147 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:11:01.653021 31872 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786334794676589-31872-0/minicluster-data/master-2/wal/instance
I20260810 04:11:01.653231 31872 external_mini_cluster.cc:1428] Setting key bb1942ea1fa4911bb8452043b46bc0f5
I20260810 04:11:01.655495 24147 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.32.62" port: 35493 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 04:11:01 dist-test-slave-8w9v krb5kdc[23885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335061, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.32.60@KRBTEST.COM
Aug 10 04:11:01 dist-test-slave-8w9v krb5kdc[23885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335061, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.60@KRBTEST.COM for kudu/127.31.32.62@KRBTEST.COM
I20260810 04:11:01.664021 23976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:11:01.656221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38292 (local address 127.31.32.62:35493)
0810 04:11:01.656313 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0810 04:11:01.656315 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:11:01.656443 (+ 128us) server_negotiation.cc:407] Connection header received
0810 04:11:01.656515 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:11:01.656516 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:11:01.656526 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:11:01.656572 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:11:01.657065 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:11:01.657667 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:11:01.658428 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:11:01.658634 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:11:01.658717 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:11:01.661085 (+ 2368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:11:01.661086 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:11:01.661086 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:11:01.661093 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:11:01.662569 (+ 1476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:11:01.662993 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:11:01.662994 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:11:01.662994 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:11:01.663076 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:11:01.663390 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:11:01.663391 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:11:01.663391 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:11:01.663513 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:11:01.663583 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0810 04:11:01.663834 (+ 251us) server_negotiation.cc:299] Negotiation successful
0810 04:11:01.663917 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260810 04:11:01.664250 24150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:11:01.653908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58826 (local address 127.31.32.60:37165)
0810 04:11:01.654062 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0810 04:11:01.654070 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 04:11:01.654094 (+ 24us) server_negotiation.cc:407] Connection header received
0810 04:11:01.654550 (+ 456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:11:01.654557 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:11:01.654611 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:11:01.654721 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:11:01.655567 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:11:01.656843 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:11:01.657652 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:11:01.658088 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:11:01.658235 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:11:01.660620 (+ 2385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:11:01.660628 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:11:01.660628 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:11:01.660634 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:11:01.662474 (+ 1840us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:11:01.662990 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:11:01.662994 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:11:01.662994 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:11:01.663114 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:11:01.663324 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:11:01.663324 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:11:01.663325 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:11:01.663611 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:11:01.663750 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0810 04:11:01.663892 (+ 142us) server_negotiation.cc:299] Negotiation successful
0810 04:11:01.664152 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":58,"server-negotiator.queue_time_us":112,"thread_start_us":70,"threads_started":1}
I20260810 04:11:01.664255 24152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:11:01.656165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:35493 (local address 127.0.0.1:38292)
0810 04:11:01.656337 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0810 04:11:01.656376 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0810 04:11:01.656462 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:11:01.656612 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:11:01.656615 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:11:01.656900 (+ 285us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:11:01.656964 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:11:01.656966 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:11:01.657737 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:11:01.657739 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:11:01.658334 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:11:01.658336 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:11:01.658754 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:11:01.658755 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:11:01.658826 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:11:01.659260 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:11:01.659268 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:11:01.661000 (+ 1732us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:11:01.662642 (+ 1642us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:11:01.662650 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:11:01.662652 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:11:01.662905 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:11:01.663182 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:11:01.663182 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:11:01.663183 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:11:01.663321 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:11:01.663583 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:11:01.663588 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:11:01.663643 (+ 55us) client_negotiation.cc:769] Sending connection context
0810 04:11:01.663795 (+ 152us) client_negotiation.cc:240] Negotiation successful
0810 04:11:01.663973 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"mutex_wait_us":1,"thread_start_us":94,"threads_started":1}
I20260810 04:11:01.664949 24147 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.32.61" port: 45243 } has no permanent_uuid. Determining permanent_uuid...
I20260810 04:11:01.666119 23973 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.32.60" port: 37165 } attempt: 2
Aug 10 04:11:01 dist-test-slave-8w9v krb5kdc[23885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335061, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.60@KRBTEST.COM for kudu/127.31.32.61@KRBTEST.COM
Aug 10 04:11:01 dist-test-slave-8w9v krb5kdc[23885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335061, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.32.62@KRBTEST.COM
Aug 10 04:11:01 dist-test-slave-8w9v krb5kdc[23885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335061, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.62@KRBTEST.COM for kudu/127.31.32.60@KRBTEST.COM
I20260810 04:11:01.672894 24152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:11:01.665178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.61:45243 (local address 127.0.0.1:56536)
0810 04:11:01.665223 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0810 04:11:01.665232 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 04:11:01.665265 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:11:01.665411 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:11:01.665411 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:11:01.665544 (+ 133us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:11:01.665570 (+ 26us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:11:01.665571 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:11:01.666168 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:11:01.666170 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:11:01.666908 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:11:01.666910 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:11:01.667391 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:11:01.667392 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:11:01.667451 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:11:01.667893 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:11:01.667898 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:11:01.669218 (+ 1320us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:11:01.671669 (+ 2451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:11:01.671670 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:11:01.671671 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:11:01.671879 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:11:01.672146 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:11:01.672147 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:11:01.672147 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:11:01.672245 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:11:01.672542 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:11:01.672543 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:11:01.672584 (+ 41us) client_negotiation.cc:769] Sending connection context
0810 04:11:01.672674 (+ 90us) client_negotiation.cc:240] Negotiation successful
0810 04:11:01.672800 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260810 04:11:01.672894 24069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:11:01.665256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56536 (local address 127.31.32.61:45243)
0810 04:11:01.665308 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0810 04:11:01.665310 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:11:01.665319 (+ 9us) server_negotiation.cc:407] Connection header received
0810 04:11:01.665342 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:11:01.665343 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:11:01.665352 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:11:01.665404 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:11:01.665645 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:11:01.666089 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:11:01.666990 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:11:01.667314 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:11:01.667392 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:11:01.669332 (+ 1940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:11:01.669333 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:11:01.669334 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:11:01.669341 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:11:01.671026 (+ 1685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:11:01.671950 (+ 924us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:11:01.671952 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:11:01.671952 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:11:01.672020 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:11:01.672345 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:11:01.672346 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:11:01.672346 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:11:01.672463 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:11:01.672537 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0810 04:11:01.672702 (+ 165us) server_negotiation.cc:299] Negotiation successful
0810 04:11:01.672785 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260810 04:11:01.673254 23976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:11:01.665069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38298 (local address 127.31.32.62:35493)
0810 04:11:01.665232 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0810 04:11:01.665234 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:11:01.665241 (+ 7us) server_negotiation.cc:407] Connection header received
0810 04:11:01.665259 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:11:01.665260 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:11:01.665268 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:11:01.665338 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:11:01.665785 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:11:01.666278 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:11:01.667250 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:11:01.667566 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:11:01.667629 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:11:01.670171 (+ 2542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:11:01.670172 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:11:01.670172 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:11:01.670179 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:11:01.671568 (+ 1389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:11:01.671968 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:11:01.671969 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:11:01.671969 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:11:01.672083 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:11:01.672321 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:11:01.672322 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:11:01.672322 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:11:01.672461 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:11:01.672632 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0810 04:11:01.673025 (+ 393us) server_negotiation.cc:299] Negotiation successful
0810 04:11:01.673135 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":28,"server-negotiator.queue_time_us":145}
I20260810 04:11:01.673760 24147 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.32.60" port: 37165 } has no permanent_uuid. Determining permanent_uuid...
I20260810 04:11:01.674109 23975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:11:01.666335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.60:37165 (local address 127.0.0.1:58838)
0810 04:11:01.666395 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0810 04:11:01.666404 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 04:11:01.666435 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:11:01.666586 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:11:01.666587 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:11:01.666735 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:11:01.666769 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:11:01.666770 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:11:01.667548 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:11:01.667549 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:11:01.668406 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:11:01.668409 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:11:01.668979 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:11:01.668980 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:11:01.669051 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:11:01.669432 (+ 381us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:11:01.669436 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:11:01.670985 (+ 1549us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:11:01.672948 (+ 1963us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:11:01.672949 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:11:01.672949 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:11:01.673178 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:11:01.673420 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:11:01.673421 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:11:01.673421 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:11:01.673499 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:11:01.673765 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:11:01.673766 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:11:01.673822 (+ 56us) client_negotiation.cc:769] Sending connection context
0810 04:11:01.673894 (+ 72us) client_negotiation.cc:240] Negotiation successful
0810 04:11:01.673993 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47,"mutex_wait_us":17}
I20260810 04:11:01.674427 24150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:11:01.666390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58838 (local address 127.31.32.60:37165)
0810 04:11:01.666436 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0810 04:11:01.666438 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:11:01.666445 (+ 7us) server_negotiation.cc:407] Connection header received
0810 04:11:01.666486 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:11:01.666486 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:11:01.666494 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:11:01.666581 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:11:01.666855 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:11:01.667466 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:11:01.668498 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:11:01.668873 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:11:01.669184 (+ 311us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:11:01.671079 (+ 1895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:11:01.671080 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:11:01.671080 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:11:01.671087 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:11:01.672861 (+ 1774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:11:01.673264 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:11:01.673265 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:11:01.673266 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:11:01.673337 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:11:01.673576 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:11:01.673577 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:11:01.673577 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:11:01.673705 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:11:01.673773 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0810 04:11:01.674227 (+ 454us) server_negotiation.cc:299] Negotiation successful
0810 04:11:01.674320 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260810 04:11:01.676561 23973 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14543a50c1dd47ee994208d8f3c5d27f: Bootstrap starting.
I20260810 04:11:01.677400 23973 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 14543a50c1dd47ee994208d8f3c5d27f: Neither blocks nor log segments found. Creating new log.
I20260810 04:11:01.677762 23973 log.cc:824] T 00000000000000000000000000000000 P 14543a50c1dd47ee994208d8f3c5d27f: Log is configured to *not* fsync() on all Append() calls
Aug 10 04:11:01 dist-test-slave-8w9v krb5kdc[23885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335061, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.32.61@KRBTEST.COM
I20260810 04:11:01.678582 23973 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14543a50c1dd47ee994208d8f3c5d27f: No bootstrap required, opened a new log
Aug 10 04:11:01 dist-test-slave-8w9v krb5kdc[23885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335061, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.60@KRBTEST.COM for kudu/127.31.32.60@KRBTEST.COM
I20260810 04:11:01.680959 23973 raft_consensus.cc:348] T 00000000000000000000000000000000 P 14543a50c1dd47ee994208d8f3c5d27f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14543a50c1dd47ee994208d8f3c5d27f" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 35493 } } peers { permanent_uuid: "40a0dbb48ab54d009a6e59de7a258bd6" member_type: VOTER last_known_addr { host: "127.31.32.61" port: 45243 } } peers { permanent_uuid: "0c3ee222ef2949c1ae46261b3f9114af" member_type: VOTER last_known_addr { host: "127.31.32.60" port: 37165 } }
I20260810 04:11:01.681113 23973 raft_consensus.cc:729] T 00000000000000000000000000000000 P 14543a50c1dd47ee994208d8f3c5d27f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14543a50c1dd47ee994208d8f3c5d27f, State: Initialized, Role: FOLLOWER
I20260810 04:11:01.681236 23973 consensus_queue.cc:260] T 00000000000000000000000000000000 P 14543a50c1dd47ee994208d8f3c5d27f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14543a50c1dd47ee994208d8f3c5d27f" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 35493 } } peers { permanent_uuid: "40a0dbb48ab54d009a6e59de7a258bd6" member_type: VOTER last_known_addr { host: "127.31.32.61" port: 45243 } } peers { permanent_uuid: "0c3ee222ef2949c1ae46261b3f9114af" member_type: VOTER last_known_addr { host: "127.31.32.60" port: 37165 } }
I20260810 04:11:01.681604 23973 sys_catalog.cc:564] T 00000000000000000000000000000000 P 14543a50c1dd47ee994208d8f3c5d27f [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:11:01.681651 24069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:11:01.673966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56546 (local address 127.31.32.61:45243)
0810 04:11:01.674000 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0810 04:11:01.674002 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:11:01.674009 (+ 7us) server_negotiation.cc:407] Connection header received
0810 04:11:01.674027 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:11:01.674028 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:11:01.674036 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:11:01.674086 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:11:01.674751 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:11:01.675295 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:11:01.675935 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:11:01.676234 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:11:01.676300 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:11:01.678720 (+ 2420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:11:01.678721 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:11:01.678722 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:11:01.678728 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:11:01.680332 (+ 1604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:11:01.680675 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:11:01.680676 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:11:01.680676 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:11:01.680734 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:11:01.680949 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:11:01.680951 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:11:01.680951 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:11:01.681083 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:11:01.681241 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0810 04:11:01.681475 (+ 234us) server_negotiation.cc:299] Negotiation successful
0810 04:11:01.681560 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260810 04:11:01.682085 24156 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14543a50c1dd47ee994208d8f3c5d27f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14543a50c1dd47ee994208d8f3c5d27f" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 35493 } } peers { permanent_uuid: "40a0dbb48ab54d009a6e59de7a258bd6" member_type: VOTER last_known_addr { host: "127.31.32.61" port: 45243 } } peers { permanent_uuid: "0c3ee222ef2949c1ae46261b3f9114af" member_type: VOTER last_known_addr { host: "127.31.32.60" port: 37165 } } }
I20260810 04:11:01.682276 24156 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14543a50c1dd47ee994208d8f3c5d27f [sys.catalog]: This master's current role is: FOLLOWER
I20260810 04:11:01.683018 24152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:11:01.673983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.60:37165 (local address 127.0.0.1:58852)
0810 04:11:01.674054 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0810 04:11:01.674062 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0810 04:11:01.674094 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:11:01.674636 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:11:01.674637 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:11:01.674785 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:11:01.674825 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:11:01.674827 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:11:01.675600 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:11:01.675601 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:11:01.676349 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:11:01.676352 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:11:01.676788 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:11:01.676789 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:11:01.676828 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:11:01.677242 (+ 414us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:11:01.677247 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:11:01.679132 (+ 1885us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:11:01.681930 (+ 2798us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:11:01.681931 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:11:01.681932 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:11:01.682176 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:11:01.682377 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:11:01.682378 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:11:01.682378 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:11:01.682437 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:11:01.682727 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:11:01.682728 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:11:01.682765 (+ 37us) client_negotiation.cc:769] Sending connection context
0810 04:11:01.682829 (+ 64us) client_negotiation.cc:240] Negotiation successful
0810 04:11:01.682920 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260810 04:11:01.683244 31872 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 04:11:01.683383 24154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:11:01.674053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58852 (local address 127.31.32.60:37165)
0810 04:11:01.674407 (+ 354us) server_negotiation.cc:206] Beginning negotiation
0810 04:11:01.674409 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:11:01.674415 (+ 6us) server_negotiation.cc:407] Connection header received
0810 04:11:01.674429 (+ 14us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:11:01.674430 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:11:01.674437 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:11:01.674632 (+ 195us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:11:01.674884 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:11:01.675511 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:11:01.676440 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:11:01.676711 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:11:01.676971 (+ 260us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:11:01.679203 (+ 2232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:11:01.679204 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:11:01.679204 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:11:01.679212 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:11:01.681866 (+ 2654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:11:01.682253 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:11:01.682254 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:11:01.682254 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:11:01.682312 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:11:01.682505 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:11:01.682505 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:11:01.682506 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:11:01.682653 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:11:01.683132 (+ 479us) server_negotiation.cc:1035] Waiting for connection context
0810 04:11:01.683172 (+ 40us) server_negotiation.cc:299] Negotiation successful
0810 04:11:01.683265 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":102,"threads_started":1}
W20260810 04:11:01.684085 24170 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 14543a50c1dd47ee994208d8f3c5d27f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 04:11:01.684213 24170 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 04:11:01.684747 24147 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c3ee222ef2949c1ae46261b3f9114af: Bootstrap starting.
I20260810 04:11:01.685354 24147 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0c3ee222ef2949c1ae46261b3f9114af: Neither blocks nor log segments found. Creating new log.
I20260810 04:11:01.685644 24147 log.cc:824] T 00000000000000000000000000000000 P 0c3ee222ef2949c1ae46261b3f9114af: Log is configured to *not* fsync() on all Append() calls
I20260810 04:11:01.686339 24147 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c3ee222ef2949c1ae46261b3f9114af: No bootstrap required, opened a new log
I20260810 04:11:01.687685 24147 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0c3ee222ef2949c1ae46261b3f9114af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14543a50c1dd47ee994208d8f3c5d27f" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 35493 } } peers { permanent_uuid: "40a0dbb48ab54d009a6e59de7a258bd6" member_type: VOTER last_known_addr { host: "127.31.32.61" port: 45243 } } peers { permanent_uuid: "0c3ee222ef2949c1ae46261b3f9114af" member_type: VOTER last_known_addr { host: "127.31.32.60" port: 37165 } }
I20260810 04:11:01.687832 24147 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0c3ee222ef2949c1ae46261b3f9114af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c3ee222ef2949c1ae46261b3f9114af, State: Initialized, Role: FOLLOWER
I20260810 04:11:01.684981 23900 master_runner.cc:186] Error getting master registration for 127.31.32.61:45243: OK, instance_id { permanent_uuid: "40a0dbb48ab54d009a6e59de7a258bd6" instance_seqno: 1786335061470787 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 04:11:01.687918 23900 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 04:11:01.687927 24147 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0c3ee222ef2949c1ae46261b3f9114af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14543a50c1dd47ee994208d8f3c5d27f" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 35493 } } peers { permanent_uuid: "40a0dbb48ab54d009a6e59de7a258bd6" member_type: VOTER last_known_addr { host: "127.31.32.61" port: 45243 } } peers { permanent_uuid: "0c3ee222ef2949c1ae46261b3f9114af" member_type: VOTER last_known_addr { host: "127.31.32.60" port: 37165 } }
I20260810 04:11:01.688220 23946 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "14543a50c1dd47ee994208d8f3c5d27f"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38298
I20260810 04:11:01.688323 24147 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0c3ee222ef2949c1ae46261b3f9114af [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:11:01.688278 24173 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c3ee222ef2949c1ae46261b3f9114af [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14543a50c1dd47ee994208d8f3c5d27f" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 35493 } } peers { permanent_uuid: "40a0dbb48ab54d009a6e59de7a258bd6" member_type: VOTER last_known_addr { host: "127.31.32.61" port: 45243 } } peers { permanent_uuid: "0c3ee222ef2949c1ae46261b3f9114af" member_type: VOTER last_known_addr { host: "127.31.32.60" port: 37165 } } }
I20260810 04:11:01.688344 23946 raft_consensus.cc:482] T 00000000000000000000000000000000 P 14543a50c1dd47ee994208d8f3c5d27f [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260810 04:11:01.688369 24173 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c3ee222ef2949c1ae46261b3f9114af [sys.catalog]: This master's current role is: FOLLOWER
I20260810 04:11:01.688390 23946 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 14543a50c1dd47ee994208d8f3c5d27f [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:11:01.689263 23946 raft_consensus.cc:504] T 00000000000000000000000000000000 P 14543a50c1dd47ee994208d8f3c5d27f [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14543a50c1dd47ee994208d8f3c5d27f" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 35493 } } peers { permanent_uuid: "40a0dbb48ab54d009a6e59de7a258bd6" member_type: VOTER last_known_addr { host: "127.31.32.61" port: 45243 } } peers { permanent_uuid: "0c3ee222ef2949c1ae46261b3f9114af" member_type: VOTER last_known_addr { host: "127.31.32.60" port: 37165 } }
I20260810 04:11:01.689489 23946 leader_election.cc:288] T 00000000000000000000000000000000 P 14543a50c1dd47ee994208d8f3c5d27f [CANDIDATE]: Term 1 election: Requested vote from peers 40a0dbb48ab54d009a6e59de7a258bd6 (127.31.32.61:45243), 0c3ee222ef2949c1ae46261b3f9114af (127.31.32.60:37165)
I20260810 04:11:01.689864 24119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "14543a50c1dd47ee994208d8f3c5d27f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "0c3ee222ef2949c1ae46261b3f9114af"
I20260810 04:11:01.689962 24119 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0c3ee222ef2949c1ae46261b3f9114af [term 0 FOLLOWER]: Advancing to term 1
W20260810 04:11:01.690702 24185 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0c3ee222ef2949c1ae46261b3f9114af: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 04:11:01.690771 24185 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 04:11:01.691268 24119 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0c3ee222ef2949c1ae46261b3f9114af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14543a50c1dd47ee994208d8f3c5d27f in term 1.
I20260810 04:11:01.691538 23923 leader_election.cc:302] T 00000000000000000000000000000000 P 14543a50c1dd47ee994208d8f3c5d27f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c3ee222ef2949c1ae46261b3f9114af, 14543a50c1dd47ee994208d8f3c5d27f; no voters:
I20260810 04:11:01.691648 24156 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 14543a50c1dd47ee994208d8f3c5d27f [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:11:01.691843 24156 raft_consensus.cc:686] T 00000000000000000000000000000000 P 14543a50c1dd47ee994208d8f3c5d27f [term 1 LEADER]: Becoming Leader. State: Replica: 14543a50c1dd47ee994208d8f3c5d27f, State: Running, Role: LEADER
I20260810 04:11:01.691936 24156 consensus_queue.cc:237] T 00000000000000000000000000000000 P 14543a50c1dd47ee994208d8f3c5d27f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14543a50c1dd47ee994208d8f3c5d27f" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 35493 } } peers { permanent_uuid: "40a0dbb48ab54d009a6e59de7a258bd6" member_type: VOTER last_known_addr { host: "127.31.32.61" port: 45243 } } peers { permanent_uuid: "0c3ee222ef2949c1ae46261b3f9114af" member_type: VOTER last_known_addr { host: "127.31.32.60" port: 37165 } }
I20260810 04:11:01.692759 24195 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14543a50c1dd47ee994208d8f3c5d27f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 14543a50c1dd47ee994208d8f3c5d27f. Latest consensus state: current_term: 1 leader_uuid: "14543a50c1dd47ee994208d8f3c5d27f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14543a50c1dd47ee994208d8f3c5d27f" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 35493 } } peers { permanent_uuid: "40a0dbb48ab54d009a6e59de7a258bd6" member_type: VOTER last_known_addr { host: "127.31.32.61" port: 45243 } } peers { permanent_uuid: "0c3ee222ef2949c1ae46261b3f9114af" member_type: VOTER last_known_addr { host: "127.31.32.60" port: 37165 } } }
I20260810 04:11:01.692972 24195 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14543a50c1dd47ee994208d8f3c5d27f [sys.catalog]: This master's current role is: LEADER
I20260810 04:11:01.693588 24073 master_runner.cc:186] Error getting master registration for 127.31.32.61:45243: OK, instance_id { permanent_uuid: "40a0dbb48ab54d009a6e59de7a258bd6" instance_seqno: 1786335061470787 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 04:11:01.695801 24073 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 04:11:01.695962 24198 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 04:11:01.696223 24198 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 04:11:01.697975 24119 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0c3ee222ef2949c1ae46261b3f9114af [term 1 FOLLOWER]: Refusing update from remote peer 14543a50c1dd47ee994208d8f3c5d27f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:11:01.698275 24156 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 14543a50c1dd47ee994208d8f3c5d27f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c3ee222ef2949c1ae46261b3f9114af" member_type: VOTER last_known_addr { host: "127.31.32.60" port: 37165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:11:01.699429 24173 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c3ee222ef2949c1ae46261b3f9114af [sys.catalog]: SysCatalogTable state changed. Reason: New leader 14543a50c1dd47ee994208d8f3c5d27f. Latest consensus state: current_term: 1 leader_uuid: "14543a50c1dd47ee994208d8f3c5d27f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14543a50c1dd47ee994208d8f3c5d27f" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 35493 } } peers { permanent_uuid: "40a0dbb48ab54d009a6e59de7a258bd6" member_type: VOTER last_known_addr { host: "127.31.32.61" port: 45243 } } peers { permanent_uuid: "0c3ee222ef2949c1ae46261b3f9114af" member_type: VOTER last_known_addr { host: "127.31.32.60" port: 37165 } } }
I20260810 04:11:01.699527 24173 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c3ee222ef2949c1ae46261b3f9114af [sys.catalog]: This master's current role is: FOLLOWER
I20260810 04:11:01.703008 23976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:11:01.691093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38300 (local address 127.31.32.62:35493)
0810 04:11:01.691135 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0810 04:11:01.691137 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:11:01.693815 (+ 2678us) server_negotiation.cc:407] Connection header received
0810 04:11:01.693916 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:11:01.693917 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:11:01.693927 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:11:01.693958 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:11:01.694753 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:11:01.695484 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:11:01.698087 (+ 2603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:11:01.698433 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:11:01.698477 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:11:01.700071 (+ 1594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:11:01.700073 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:11:01.700073 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:11:01.700081 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:11:01.701778 (+ 1697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:11:01.702184 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:11:01.702185 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:11:01.702186 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:11:01.702239 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:11:01.702448 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:11:01.702449 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:11:01.702449 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:11:01.702569 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:11:01.702648 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0810 04:11:01.702845 (+ 197us) server_negotiation.cc:299] Negotiation successful
0810 04:11:01.702920 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260810 04:11:01.704737 24195 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14543a50c1dd47ee994208d8f3c5d27f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "14543a50c1dd47ee994208d8f3c5d27f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14543a50c1dd47ee994208d8f3c5d27f" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 35493 } } peers { permanent_uuid: "40a0dbb48ab54d009a6e59de7a258bd6" member_type: VOTER last_known_addr { host: "127.31.32.61" port: 45243 } } peers { permanent_uuid: "0c3ee222ef2949c1ae46261b3f9114af" member_type: VOTER last_known_addr { host: "127.31.32.60" port: 37165 } } }
I20260810 04:11:01.704828 24195 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14543a50c1dd47ee994208d8f3c5d27f [sys.catalog]: This master's current role is: LEADER
I20260810 04:11:01.705173 24173 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c3ee222ef2949c1ae46261b3f9114af [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "14543a50c1dd47ee994208d8f3c5d27f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14543a50c1dd47ee994208d8f3c5d27f" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 35493 } } peers { permanent_uuid: "40a0dbb48ab54d009a6e59de7a258bd6" member_type: VOTER last_known_addr { host: "127.31.32.61" port: 45243 } } peers { permanent_uuid: "0c3ee222ef2949c1ae46261b3f9114af" member_type: VOTER last_known_addr { host: "127.31.32.60" port: 37165 } } }
I20260810 04:11:01.705250 24173 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c3ee222ef2949c1ae46261b3f9114af [sys.catalog]: This master's current role is: FOLLOWER
I20260810 04:11:01.705579 24198 catalog_manager.cc:1347] Generated new cluster ID: b59153eab5094379ad2667ab05d07522
I20260810 04:11:01.705646 24198 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 04:11:01.707646 24065 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.32.60" port: 37165 } attempt: 3
Aug 10 04:11:01 dist-test-slave-8w9v krb5kdc[23885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335061, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.61@KRBTEST.COM for kudu/127.31.32.60@KRBTEST.COM
I20260810 04:11:01.713006 24069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:11:01.691387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56556 (local address 127.31.32.61:45243)
0810 04:11:01.691429 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0810 04:11:01.691431 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:11:01.697897 (+ 6466us) server_negotiation.cc:407] Connection header received
0810 04:11:01.697996 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:11:01.697997 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:11:01.698007 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:11:01.698038 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:11:01.699902 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:11:01.700653 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:11:01.701621 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:11:01.702359 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:11:01.702426 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:11:01.706474 (+ 4048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:11:01.706475 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:11:01.706475 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:11:01.706482 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:11:01.708209 (+ 1727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:11:01.708599 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:11:01.708600 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:11:01.708601 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:11:01.708658 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:11:01.712358 (+ 3700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:11:01.712359 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:11:01.712359 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:11:01.712494 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:11:01.712578 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0810 04:11:01.712822 (+ 244us) server_negotiation.cc:299] Negotiation successful
0810 04:11:01.712895 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260810 04:11:01.705468 24202 mvcc.cc:205] Tried to move back new op lower bound from 7316828412712292352 to 7316828412692205568. Current Snapshot: MvccSnapshot[applied={T|T < 7316828412712292352}]
I20260810 04:11:01.713414 24201 mvcc.cc:205] Tried to move back new op lower bound from 7316828412712292352 to 7316828412692205568. Current Snapshot: MvccSnapshot[applied={T|T < 7316828412712292352 or (T in {7316828412712292352})}]
I20260810 04:11:01.714267 24154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:11:01.692281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58862 (local address 127.31.32.60:37165)
0810 04:11:01.692347 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0810 04:11:01.692349 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:11:01.699989 (+ 7640us) server_negotiation.cc:407] Connection header received
0810 04:11:01.700077 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:11:01.700078 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:11:01.700087 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:11:01.700144 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:11:01.701642 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:11:01.702409 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:11:01.704664 (+ 2255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:11:01.704970 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:11:01.705016 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:11:01.706462 (+ 1446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:11:01.706463 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:11:01.706463 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:11:01.706484 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:11:01.712841 (+ 6357us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:11:01.713286 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:11:01.713287 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:11:01.713287 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:11:01.713356 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:11:01.713628 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:11:01.713629 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:11:01.713629 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:11:01.713765 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:11:01.713846 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0810 04:11:01.714059 (+ 213us) server_negotiation.cc:299] Negotiation successful
0810 04:11:01.714174 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260810 04:11:01.716254 24150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:11:01.707966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58868 (local address 127.31.32.60:37165)
0810 04:11:01.708011 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0810 04:11:01.708013 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:11:01.708023 (+ 10us) server_negotiation.cc:407] Connection header received
0810 04:11:01.708043 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:11:01.708044 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:11:01.708052 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:11:01.708138 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:11:01.708430 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:11:01.709101 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:11:01.710102 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:11:01.710373 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:11:01.710434 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:11:01.713566 (+ 3132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:11:01.713567 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:11:01.713568 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:11:01.713574 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:11:01.715119 (+ 1545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:11:01.715474 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:11:01.715475 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:11:01.715476 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:11:01.715522 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:11:01.715741 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:11:01.715742 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:11:01.715742 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:11:01.715854 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:11:01.715908 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0810 04:11:01.716089 (+ 181us) server_negotiation.cc:299] Negotiation successful
0810 04:11:01.716177 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260810 04:11:01.716279 24067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:11:01.707892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.60:37165 (local address 127.0.0.1:58868)
0810 04:11:01.707940 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0810 04:11:01.707950 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 04:11:01.707986 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:11:01.708151 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:11:01.708152 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:11:01.708313 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:11:01.708356 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:11:01.708358 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:11:01.709181 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:11:01.709182 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:11:01.710004 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:11:01.710007 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:11:01.710439 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:11:01.710440 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:11:01.710496 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:11:01.711079 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:11:01.711084 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:11:01.713484 (+ 2400us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:11:01.715200 (+ 1716us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:11:01.715201 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:11:01.715202 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:11:01.715405 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:11:01.715587 (+ 182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:11:01.715588 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:11:01.715588 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:11:01.715673 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:11:01.715919 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:11:01.715919 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:11:01.715975 (+ 56us) client_negotiation.cc:769] Sending connection context
0810 04:11:01.716067 (+ 92us) client_negotiation.cc:240] Negotiation successful
0810 04:11:01.716185 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260810 04:11:01.718359 24065 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40a0dbb48ab54d009a6e59de7a258bd6: Bootstrap starting.
I20260810 04:11:01.719071 24065 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 40a0dbb48ab54d009a6e59de7a258bd6: Neither blocks nor log segments found. Creating new log.
I20260810 04:11:01.719391 24065 log.cc:824] T 00000000000000000000000000000000 P 40a0dbb48ab54d009a6e59de7a258bd6: Log is configured to *not* fsync() on all Append() calls
I20260810 04:11:01.720465 24065 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40a0dbb48ab54d009a6e59de7a258bd6: No bootstrap required, opened a new log
I20260810 04:11:01.722493 24065 raft_consensus.cc:348] T 00000000000000000000000000000000 P 40a0dbb48ab54d009a6e59de7a258bd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14543a50c1dd47ee994208d8f3c5d27f" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 35493 } } peers { permanent_uuid: "40a0dbb48ab54d009a6e59de7a258bd6" member_type: VOTER last_known_addr { host: "127.31.32.61" port: 45243 } } peers { permanent_uuid: "0c3ee222ef2949c1ae46261b3f9114af" member_type: VOTER last_known_addr { host: "127.31.32.60" port: 37165 } }
I20260810 04:11:01.722631 24065 raft_consensus.cc:729] T 00000000000000000000000000000000 P 40a0dbb48ab54d009a6e59de7a258bd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40a0dbb48ab54d009a6e59de7a258bd6, State: Initialized, Role: FOLLOWER
I20260810 04:11:01.722711 24065 consensus_queue.cc:260] T 00000000000000000000000000000000 P 40a0dbb48ab54d009a6e59de7a258bd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14543a50c1dd47ee994208d8f3c5d27f" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 35493 } } peers { permanent_uuid: "40a0dbb48ab54d009a6e59de7a258bd6" member_type: VOTER last_known_addr { host: "127.31.32.61" port: 45243 } } peers { permanent_uuid: "0c3ee222ef2949c1ae46261b3f9114af" member_type: VOTER last_known_addr { host: "127.31.32.60" port: 37165 } }
I20260810 04:11:01.723026 24207 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40a0dbb48ab54d009a6e59de7a258bd6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14543a50c1dd47ee994208d8f3c5d27f" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 35493 } } peers { permanent_uuid: "40a0dbb48ab54d009a6e59de7a258bd6" member_type: VOTER last_known_addr { host: "127.31.32.61" port: 45243 } } peers { permanent_uuid: "0c3ee222ef2949c1ae46261b3f9114af" member_type: VOTER last_known_addr { host: "127.31.32.60" port: 37165 } } }
I20260810 04:11:01.723291 24207 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40a0dbb48ab54d009a6e59de7a258bd6 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 04:11:01.723065 24032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "14543a50c1dd47ee994208d8f3c5d27f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "40a0dbb48ab54d009a6e59de7a258bd6"
I20260810 04:11:01.723129 24031 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 40a0dbb48ab54d009a6e59de7a258bd6 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:11:01.723033 24065 sys_catalog.cc:564] T 00000000000000000000000000000000 P 40a0dbb48ab54d009a6e59de7a258bd6 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:11:01.724612 24031 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 40a0dbb48ab54d009a6e59de7a258bd6 [term 1 FOLLOWER]: Refusing update from remote peer 14543a50c1dd47ee994208d8f3c5d27f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:11:01.724923 24195 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 14543a50c1dd47ee994208d8f3c5d27f [LEADER]: Connected to new peer: Peer: permanent_uuid: "40a0dbb48ab54d009a6e59de7a258bd6" member_type: VOTER last_known_addr { host: "127.31.32.61" port: 45243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:11:01.725914 24032 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 40a0dbb48ab54d009a6e59de7a258bd6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14543a50c1dd47ee994208d8f3c5d27f in term 1.
I20260810 04:11:01.728276 24207 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40a0dbb48ab54d009a6e59de7a258bd6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 14543a50c1dd47ee994208d8f3c5d27f. Latest consensus state: current_term: 1 leader_uuid: "14543a50c1dd47ee994208d8f3c5d27f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14543a50c1dd47ee994208d8f3c5d27f" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 35493 } } peers { permanent_uuid: "40a0dbb48ab54d009a6e59de7a258bd6" member_type: VOTER last_known_addr { host: "127.31.32.61" port: 45243 } } peers { permanent_uuid: "0c3ee222ef2949c1ae46261b3f9114af" member_type: VOTER last_known_addr { host: "127.31.32.60" port: 37165 } } }
I20260810 04:11:01.728376 24207 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40a0dbb48ab54d009a6e59de7a258bd6 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 04:11:01.728417 24207 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40a0dbb48ab54d009a6e59de7a258bd6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "14543a50c1dd47ee994208d8f3c5d27f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14543a50c1dd47ee994208d8f3c5d27f" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 35493 } } peers { permanent_uuid: "40a0dbb48ab54d009a6e59de7a258bd6" member_type: VOTER last_known_addr { host: "127.31.32.61" port: 45243 } } peers { permanent_uuid: "0c3ee222ef2949c1ae46261b3f9114af" member_type: VOTER last_known_addr { host: "127.31.32.60" port: 37165 } } }
I20260810 04:11:01.728466 24207 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40a0dbb48ab54d009a6e59de7a258bd6 [sys.catalog]: This master's current role is: FOLLOWER
W20260810 04:11:01.728987 24221 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 40a0dbb48ab54d009a6e59de7a258bd6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 04:11:01.729043 24221 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 04:11:01.730082 24218 mvcc.cc:205] Tried to move back new op lower bound from 7316828412712292352 to 7316828412692205568. Current Snapshot: MvccSnapshot[applied={T|T < 7316828412712292352}]
I20260810 04:11:01.750931 24195 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14543a50c1dd47ee994208d8f3c5d27f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "14543a50c1dd47ee994208d8f3c5d27f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14543a50c1dd47ee994208d8f3c5d27f" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 35493 } } peers { permanent_uuid: "40a0dbb48ab54d009a6e59de7a258bd6" member_type: VOTER last_known_addr { host: "127.31.32.61" port: 45243 } } peers { permanent_uuid: "0c3ee222ef2949c1ae46261b3f9114af" member_type: VOTER last_known_addr { host: "127.31.32.60" port: 37165 } } }
I20260810 04:11:01.751044 24195 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14543a50c1dd47ee994208d8f3c5d27f [sys.catalog]: This master's current role is: LEADER
I20260810 04:11:01.751840 24198 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 04:11:01.752646 24198 catalog_manager.cc:1504] Loading token signing keys...
I20260810 04:11:01.760427 24198 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 14543a50c1dd47ee994208d8f3c5d27f: Generated new TSK 0
I20260810 04:11:01.760608 24198 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 04:11:01.786382 24069 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:11:01.784539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56564 (local address 127.31.32.61:45243)
0810 04:11:01.784589 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0810 04:11:01.784591 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:11:01.785641 (+ 1050us) server_negotiation.cc:407] Connection header received
0810 04:11:01.785780 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:11:01.785781 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:11:01.785795 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:11:01.785832 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:11:01.786304 (+ 472us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56564: BlockingRecv error: recv got EOF from 127.0.0.1:56564 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
W20260810 04:11:01.787496 24150 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:11:01.784979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58870 (local address 127.31.32.60:37165)
0810 04:11:01.785044 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0810 04:11:01.785046 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:11:01.786633 (+ 1587us) server_negotiation.cc:407] Connection header received
0810 04:11:01.786907 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:11:01.786908 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:11:01.786920 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:11:01.786955 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:11:01.787429 (+ 474us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58870: BlockingRecv error: recv got EOF from 127.0.0.1:58870 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
W20260810 04:11:01.788283 23976 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:11:01.785341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38310 (local address 127.31.32.62:35493)
0810 04:11:01.785386 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0810 04:11:01.785388 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:11:01.786851 (+ 1463us) server_negotiation.cc:407] Connection header received
0810 04:11:01.787669 (+ 818us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:11:01.787671 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:11:01.787684 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:11:01.787721 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:11:01.788215 (+ 494us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38310: BlockingRecv error: recv got EOF from 127.0.0.1:38310 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260810 04:11:01.791054 24069 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:11:01.790262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56568 (local address 127.31.32.61:45243)
0810 04:11:01.790299 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0810 04:11:01.790301 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:11:01.790504 (+ 203us) server_negotiation.cc:407] Connection header received
0810 04:11:01.790616 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:11:01.790617 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:11:01.790637 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:11:01.790669 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:11:01.791003 (+ 334us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56568: BlockingRecv error: recv got EOF from 127.0.0.1:56568 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260810 04:11:01.794130 24069 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:11:01.793308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56584 (local address 127.31.32.61:45243)
0810 04:11:01.793345 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0810 04:11:01.793347 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:11:01.793596 (+ 249us) server_negotiation.cc:407] Connection header received
0810 04:11:01.793696 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:11:01.793697 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:11:01.793709 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:11:01.793759 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:11:01.794084 (+ 325us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56584: BlockingRecv error: recv got EOF from 127.0.0.1:56584 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260810 04:11:01.797931 24069 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:11:01.797157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56594 (local address 127.31.32.61:45243)
0810 04:11:01.797192 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0810 04:11:01.797193 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 04:11:01.797430 (+ 237us) server_negotiation.cc:407] Connection header received
0810 04:11:01.797530 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:11:01.797530 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:11:01.797540 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:11:01.797572 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:11:01.797891 (+ 319us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56594: BlockingRecv error: recv got EOF from 127.0.0.1:56594 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260810 04:11:01.803830 24069 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:11:01.802990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56600 (local address 127.31.32.61:45243)
0810 04:11:01.803028 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0810 04:11:01.803029 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 04:11:01.803282 (+ 253us) server_negotiation.cc:407] Connection header received
0810 04:11:01.803391 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:11:01.803391 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:11:01.803402 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:11:01.803435 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:11:01.803774 (+ 339us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56600: BlockingRecv error: recv got EOF from 127.0.0.1:56600 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260810 04:11:01.814203 24069 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:11:01.813100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56612 (local address 127.31.32.61:45243)
0810 04:11:01.813144 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0810 04:11:01.813146 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:11:01.813463 (+ 317us) server_negotiation.cc:407] Connection header received
0810 04:11:01.813583 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:11:01.813584 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:11:01.813597 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:11:01.813632 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:11:01.814143 (+ 511us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56612: BlockingRecv error: recv got EOF from 127.0.0.1:56612 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260810 04:11:01.832780 24069 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:11:01.831644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56622 (local address 127.31.32.61:45243)
0810 04:11:01.831694 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0810 04:11:01.831696 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:11:01.832041 (+ 345us) server_negotiation.cc:407] Connection header received
0810 04:11:01.832208 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:11:01.832210 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:11:01.832224 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:11:01.832273 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:11:01.832715 (+ 442us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56622: BlockingRecv error: recv got EOF from 127.0.0.1:56622 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
W20260810 04:11:01.867589 24069 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:11:01.866454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56630 (local address 127.31.32.61:45243)
0810 04:11:01.866503 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0810 04:11:01.866505 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:11:01.866857 (+ 352us) server_negotiation.cc:407] Connection header received
0810 04:11:01.866983 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:11:01.866985 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:11:01.866999 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:11:01.867037 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:11:01.867522 (+ 485us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56630: BlockingRecv error: recv got EOF from 127.0.0.1:56630 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
W20260810 04:11:01.933889 24069 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:11:01.933179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56642 (local address 127.31.32.61:45243)
0810 04:11:01.933300 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0810 04:11:01.933302 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:11:01.933318 (+ 16us) server_negotiation.cc:407] Connection header received
0810 04:11:01.933400 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:11:01.933401 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:11:01.933413 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:11:01.933479 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:11:01.933812 (+ 333us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56642: BlockingRecv error: recv got EOF from 127.0.0.1:56642 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260810 04:11:02.064069 24069 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:11:02.063252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56656 (local address 127.31.32.61:45243)
0810 04:11:02.063343 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0810 04:11:02.063345 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:11:02.063394 (+ 49us) server_negotiation.cc:407] Connection header received
0810 04:11:02.063466 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:11:02.063467 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:11:02.063478 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:11:02.063537 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:11:02.063974 (+ 437us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56656: BlockingRecv error: recv got EOF from 127.0.0.1:56656 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260810 04:11:02.322014 24069 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:11:02.321413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56666 (local address 127.31.32.61:45243)
0810 04:11:02.321487 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0810 04:11:02.321489 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:11:02.321500 (+ 11us) server_negotiation.cc:407] Connection header received
0810 04:11:02.321527 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:11:02.321528 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:11:02.321537 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:11:02.321600 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:11:02.321934 (+ 334us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56666: BlockingRecv error: recv got EOF from 127.0.0.1:56666 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
I20260810 04:11:02.691241 24185 catalog_manager.cc:1259] Loaded cluster ID: b59153eab5094379ad2667ab05d07522
I20260810 04:11:02.691314 24185 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0c3ee222ef2949c1ae46261b3f9114af: loading cluster ID for follower catalog manager: success
I20260810 04:11:02.692224 24185 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0c3ee222ef2949c1ae46261b3f9114af: acquiring CA information for follower catalog manager: success
I20260810 04:11:02.692557 24185 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0c3ee222ef2949c1ae46261b3f9114af: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 04:11:02.729492 24221 catalog_manager.cc:1259] Loaded cluster ID: b59153eab5094379ad2667ab05d07522
I20260810 04:11:02.729600 24221 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 40a0dbb48ab54d009a6e59de7a258bd6: loading cluster ID for follower catalog manager: success
I20260810 04:11:02.730422 24221 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 40a0dbb48ab54d009a6e59de7a258bd6: acquiring CA information for follower catalog manager: success
I20260810 04:11:02.730757 24221 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 40a0dbb48ab54d009a6e59de7a258bd6: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 04:11:02.837641 24299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:11:02.835215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56682 (local address 127.31.32.61:45243)
0810 04:11:02.835336 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0810 04:11:02.835338 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:11:02.835373 (+ 35us) server_negotiation.cc:407] Connection header received
0810 04:11:02.835399 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:11:02.835400 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:11:02.835453 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:11:02.835518 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 04:11:02.835685 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:11:02.836243 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:11:02.836930 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:11:02.837108 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:11:02.837156 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:11:02.837296 (+ 140us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 04:11:02.837420 (+ 124us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 04:11:02.837481 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0810 04:11:02.837510 (+ 29us) server_negotiation.cc:299] Negotiation successful
0810 04:11:02.837526 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85,"thread_start_us":56,"threads_started":1}
I20260810 04:11:02.841907 24307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:11:02.839709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58880 (local address 127.31.32.60:37165)
0810 04:11:02.839822 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0810 04:11:02.839824 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:11:02.839843 (+ 19us) server_negotiation.cc:407] Connection header received
0810 04:11:02.839896 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:11:02.839897 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:11:02.839933 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:11:02.839990 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 04:11:02.840090 (+ 100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:11:02.840592 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:11:02.841238 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:11:02.841418 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:11:02.841465 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:11:02.841599 (+ 134us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 04:11:02.841687 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 04:11:02.841738 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0810 04:11:02.841791 (+ 53us) server_negotiation.cc:299] Negotiation successful
0810 04:11:02.841807 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87,"thread_start_us":64,"threads_started":1}
I20260810 04:11:02.846930 24307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:11:02.843895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58890 (local address 127.31.32.60:37165)
0810 04:11:02.843961 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0810 04:11:02.843963 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:11:02.844100 (+ 137us) server_negotiation.cc:407] Connection header received
0810 04:11:02.844222 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:11:02.844223 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:11:02.844232 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:11:02.844261 (+ 29us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 04:11:02.844478 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:11:02.845151 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:11:02.845980 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:11:02.846322 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:11:02.846388 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:11:02.846538 (+ 150us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 04:11:02.846568 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 04:11:02.846620 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0810 04:11:02.846837 (+ 217us) server_negotiation.cc:299] Negotiation successful
0810 04:11:02.846858 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260810 04:11:02.847321 24314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:11:02.843614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38324 (local address 127.31.32.62:35493)
0810 04:11:02.843743 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0810 04:11:02.843745 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:11:02.843755 (+ 10us) server_negotiation.cc:407] Connection header received
0810 04:11:02.843807 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:11:02.843808 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:11:02.843858 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:11:02.843936 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 04:11:02.844223 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:11:02.845009 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:11:02.846199 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:11:02.846538 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:11:02.846616 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:11:02.846788 (+ 172us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 04:11:02.846897 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 04:11:02.846968 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0810 04:11:02.847162 (+ 194us) server_negotiation.cc:299] Negotiation successful
0810 04:11:02.847190 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":66,"threads_started":1}
I20260810 04:11:02.848109 24299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:11:02.843778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56694 (local address 127.31.32.61:45243)
0810 04:11:02.843817 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0810 04:11:02.843819 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:11:02.844106 (+ 287us) server_negotiation.cc:407] Connection header received
0810 04:11:02.844165 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:11:02.844166 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:11:02.844178 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:11:02.844227 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 04:11:02.845272 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:11:02.846008 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:11:02.847341 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:11:02.847593 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:11:02.847655 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:11:02.847810 (+ 155us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 04:11:02.847848 (+ 38us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 04:11:02.847910 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0810 04:11:02.848007 (+ 97us) server_negotiation.cc:299] Negotiation successful
0810 04:11:02.848034 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260810 04:11:02.849604 31872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskws89Io/build/release/bin/kudu with pid 23900
I20260810 04:11:02.858597 31872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskws89Io/build/release/bin/kudu with pid 23982
I20260810 04:11:02.865056 31872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskws89Io/build/release/bin/kudu with pid 24073
2026-08-10T04:11:02Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2882 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786334794676589-31872-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 04:11:02 dist-test-slave-8w9v krb5kdc[24318](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 04:11:02 dist-test-slave-8w9v krb5kdc[24318](info): set up 2 sockets
Aug 10 04:11:02 dist-test-slave-8w9v krb5kdc[24318](info): commencing operation
krb5kdc: starting...
W20260810 04:11:03.978379 31872 mini_kdc.cc:121] Time spent starting KDC: real 1.092s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786334794676589-31872-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786334794676589-31872-0/krb5kdc/test-user.keytab.
Aug 10 04:11:04 dist-test-slave-8w9v krb5kdc[24318](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335064, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T04:11:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T04:11:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.32.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.32.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.32.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.62.keytab.
Entry for principal kudu/127.31.32.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.62.keytab.
I20260810 04:11:04.107312 31872 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskws89Io/build/release/bin/kudu
/tmp/dist-test-taskws89Io/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786334794676589-31872-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786334794676589-31872-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786334794676589-31872-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786334794676589-31872-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.32.62:45203
--webserver_interface=127.31.32.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.62.keytab
--principal=kudu/127.31.32.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.32.20:43081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.32.62:45203,127.31.32.61:36501,127.31.32.60:45587
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786334794676589-31872-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786334794676589-31872-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786334794676589-31872-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:11:04.190999 24334 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 04:11:04.191215 24334 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:11:04.191233 24334 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:11:04.191253 24334 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:11:04.192431 24334 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:11:04.192476 24334 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:11:04.192492 24334 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:11:04.192504 24334 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:11:04.192516 24334 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:11:04.193956 24334 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.32.20:43081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786334794676589-31872-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786334794676589-31872-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.32.62:45203,127.31.32.61:36501,127.31.32.60:45587
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.62.keytab
--principal=kudu/127.31.32.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.32.62:45203
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786334794676589-31872-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.32.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24334
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786334794676589-31872-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:11:04.194149 24334 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:11:04.194317 24334 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:11:04.199232 24341 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:11:04.199220 24342 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:11:04.199220 24344 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:11:04.199514 24334 server_base.cc:1034] running on GCE node
I20260810 04:11:04.199793 24334 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:11:04.200114 24334 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:11:04.201334 24334 hybrid_clock.cc:648] HybridClock initialized: now 1786335064201320 us; error 41 us; skew 500 ppm
Aug 10 04:11:04 dist-test-slave-8w9v krb5kdc[24318](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335064, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:11:04.204057 24334 init.cc:377] Logged in from keytab as kudu/127.31.32.62@KRBTEST.COM (short username kudu)
I20260810 04:11:04.204607 24334 webserver.cc:460] Webserver started at http://127.31.32.62:42955/ using document root <none> and password file <none>
I20260810 04:11:04.204840 24334 fs_manager.cc:359] Metadata directory not provided
I20260810 04:11:04.204907 24334 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:11:04.205062 24334 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:11:04.206136 24334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786334794676589-31872-0/minicluster-data/master-0/data/instance:
uuid: "7834a3a484b641c88cd396050dc833a9"
format_stamp: "Formatted at 2026-08-10 04:11:04 on dist-test-slave-8w9v"
server_key: "47e0eb6afd2a47f282131d5c1501562e"
server_key_iv: "305447e44f3733e6b51497a182aaa20b"
server_key_version: "encryptionkey@0"
I20260810 04:11:04.206519 24334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786334794676589-31872-0/minicluster-data/master-0/wal/instance:
uuid: "7834a3a484b641c88cd396050dc833a9"
format_stamp: "Formatted at 2026-08-10 04:11:04 on dist-test-slave-8w9v"
server_key: "47e0eb6afd2a47f282131d5c1501562e"
server_key_iv: "305447e44f3733e6b51497a182aaa20b"
server_key_version: "encryptionkey@0"
I20260810 04:11:04.207840 24334 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 04:11:04.208658 24353 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:11:04.208832 24334 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260810 04:11:04.208931 24334 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786334794676589-31872-0/minicluster-data/master-0/data,/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786334794676589-31872-0/minicluster-data/master-0/wal
uuid: "7834a3a484b641c88cd396050dc833a9"
format_stamp: "Formatted at 2026-08-10 04:11:04 on dist-test-slave-8w9v"
server_key: "47e0eb6afd2a47f282131d5c1501562e"
server_key_iv: "305447e44f3733e6b51497a182aaa20b"
server_key_version: "encryptionkey@0"
I20260810 04:11:04.209017 24334 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786334794676589-31872-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786334794676589-31872-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786334794676589-31872-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:11:04.232406 24334 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:11:04.234871 24334 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:11:04.235031 24334 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:11:04.239367 24334 rpc_server.cc:307] RPC server started. Bound to: 127.31.32.62:45203
I20260810 04:11:04.239408 24412 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.32.62:45203 every 8 connection(s)
I20260810 04:11:04.239794 24334 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786334794676589-31872-0/minicluster-data/master-0/data/info.pb
I20260810 04:11:04.240900 24414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:11:04.242775 31872 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskws89Io/build/release/bin/kudu as pid 24334
I20260810 04:11:04.242552 24414 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.32.62" port: 45203 } has no permanent_uuid. Determining permanent_uuid...
I20260810 04:11:04.242885 31872 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786334794676589-31872-0/minicluster-data/master-0/wal/instance
I20260810 04:11:04.243072 31872 external_mini_cluster.cc:1428] Setting key 6dcac140d7006dd8a83937763f2b7c04
Aug 10 04:11:04 dist-test-slave-8w9v krb5kdc[24318](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335064, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.62@KRBTEST.COM for kudu/127.31.32.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.31.32.61@KRBTEST.COM; defaulting to no policy
I20260810 04:11:04.254457 24417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:11:04.243449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55476 (local address 127.31.32.62:45203)
0810 04:11:04.243620 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0810 04:11:04.243629 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 04:11:04.243771 (+ 142us) server_negotiation.cc:407] Connection header received
0810 04:11:04.244615 (+ 844us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:11:04.244617 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:11:04.244634 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:11:04.244729 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:11:04.245412 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:11:04.246644 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:11:04.248010 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:11:04.248392 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:11:04.248473 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:11:04.250978 (+ 2505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:11:04.250986 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:11:04.250987 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:11:04.250999 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:11:04.252731 (+ 1732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:11:04.253180 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:11:04.253183 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:11:04.253183 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:11:04.253264 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:11:04.253495 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:11:04.253496 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:11:04.253496 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:11:04.253710 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:11:04.253798 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0810 04:11:04.254000 (+ 202us) server_negotiation.cc:299] Negotiation successful
0810 04:11:04.254142 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":67,"threads_started":1}
I20260810 04:11:04.254456 24416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:11:04.243378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:45203 (local address 127.0.0.1:55476)
0810 04:11:04.243599 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0810 04:11:04.243642 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 04:11:04.244446 (+ 804us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:11:04.244767 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:11:04.244770 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:11:04.245142 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:11:04.245302 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:11:04.245307 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:11:04.246746 (+ 1439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:11:04.246748 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:11:04.247921 (+ 1173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:11:04.247925 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:11:04.248500 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:11:04.248502 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:11:04.248589 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:11:04.249138 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:11:04.249149 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:11:04.250863 (+ 1714us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:11:04.252823 (+ 1960us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:11:04.252831 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:11:04.252832 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:11:04.253044 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:11:04.253337 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:11:04.253337 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:11:04.253338 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:11:04.253428 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:11:04.253789 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:11:04.253794 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:11:04.253857 (+ 63us) client_negotiation.cc:769] Sending connection context
0810 04:11:04.253965 (+ 108us) client_negotiation.cc:240] Negotiation successful
0810 04:11:04.254242 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"mutex_wait_us":100,"thread_start_us":109,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.32.61@KRBTEST.COM" created.
I20260810 04:11:04.255393 24414 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.32.61" port: 36501 } has no permanent_uuid. Determining permanent_uuid...
I20260810 04:11:04.255755 24416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:11:04.255638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.61:36501 (local address 127.0.0.1:41190)
0810 04:11:04.255690 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0810 04:11:04.255733 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.61:36501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260810 04:11:04.255903 24357 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.32.61:36501: connect: Connection refused (error 111)
I20260810 04:11:04.256385 24416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:11:04.256279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.61:36501 (local address 127.0.0.1:41202)
0810 04:11:04.256333 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0810 04:11:04.256357 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.61:36501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260810 04:11:04.256482 24414 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.32.61:36501: Network error: Client connection negotiation failed: client connection to 127.31.32.61:36501: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.32.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.61.keytab.
Entry for principal kudu/127.31.32.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.61.keytab.
I20260810 04:11:04.270450 31872 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskws89Io/build/release/bin/kudu
/tmp/dist-test-taskws89Io/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786334794676589-31872-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786334794676589-31872-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786334794676589-31872-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786334794676589-31872-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.32.61:36501
--webserver_interface=127.31.32.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.61.keytab
--principal=kudu/127.31.32.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.32.20:43081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.32.62:45203,127.31.32.61:36501,127.31.32.60:45587
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786334794676589-31872-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786334794676589-31872-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786334794676589-31872-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 04:11:04.305747 24414 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.32.61" port: 36501 } attempt: 1
I20260810 04:11:04.306255 24416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:11:04.306111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.61:36501 (local address 127.0.0.1:41212)
0810 04:11:04.306181 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0810 04:11:04.306222 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.61:36501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260810 04:11:04.306725 24416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:11:04.306625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.61:36501 (local address 127.0.0.1:41224)
0810 04:11:04.306677 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0810 04:11:04.306704 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.61:36501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260810 04:11:04.306855 24414 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.32.61:36501: Network error: Client connection negotiation failed: client connection to 127.31.32.61:36501: connect: Connection refused (error 111)
W20260810 04:11:04.353754 24421 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 04:11:04.353965 24421 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:11:04.353983 24421 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:11:04.354002 24421 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:11:04.355140 24421 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:11:04.355197 24421 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:11:04.355214 24421 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:11:04.355227 24421 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:11:04.355240 24421 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:11:04.356818 24421 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.32.20:43081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786334794676589-31872-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786334794676589-31872-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.31.32.62:45203,127.31.32.61:36501,127.31.32.60:45587
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.61.keytab
--principal=kudu/127.31.32.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.32.61:36501
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786334794676589-31872-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.31.32.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24421
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786334794676589-31872-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:11:04.357050 24421 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:11:04.357221 24421 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:11:04.362051 24426 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:11:04.362059 24430 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:11:04.362159 24427 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:11:04.362298 24421 server_base.cc:1034] running on GCE node
I20260810 04:11:04.362551 24421 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:11:04.362882 24421 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:11:04.364041 24421 hybrid_clock.cc:648] HybridClock initialized: now 1786335064364019 us; error 45 us; skew 500 ppm
Aug 10 04:11:04 dist-test-slave-8w9v krb5kdc[24318](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335064, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:11:04.366648 24421 init.cc:377] Logged in from keytab as kudu/127.31.32.61@KRBTEST.COM (short username kudu)
I20260810 04:11:04.367115 24421 webserver.cc:460] Webserver started at http://127.31.32.61:36375/ using document root <none> and password file <none>
I20260810 04:11:04.367413 24421 fs_manager.cc:359] Metadata directory not provided
I20260810 04:11:04.367460 24421 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:11:04.367628 24421 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:11:04.368736 24421 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786334794676589-31872-0/minicluster-data/master-1/data/instance:
uuid: "50cfd406d47a44d9a1ac795c5e11b88c"
format_stamp: "Formatted at 2026-08-10 04:11:04 on dist-test-slave-8w9v"
server_key: "8147ead99a57e1e2ecaf65d00568e81d"
server_key_iv: "ee5a28d8b2750192aeb8a65d7126af75"
server_key_version: "encryptionkey@0"
I20260810 04:11:04.369091 24421 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786334794676589-31872-0/minicluster-data/master-1/wal/instance:
uuid: "50cfd406d47a44d9a1ac795c5e11b88c"
format_stamp: "Formatted at 2026-08-10 04:11:04 on dist-test-slave-8w9v"
server_key: "8147ead99a57e1e2ecaf65d00568e81d"
server_key_iv: "ee5a28d8b2750192aeb8a65d7126af75"
server_key_version: "encryptionkey@0"
I20260810 04:11:04.370426 24421 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 04:11:04.371140 24438 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:11:04.371315 24421 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 04:11:04.371433 24421 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786334794676589-31872-0/minicluster-data/master-1/data,/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786334794676589-31872-0/minicluster-data/master-1/wal
uuid: "50cfd406d47a44d9a1ac795c5e11b88c"
format_stamp: "Formatted at 2026-08-10 04:11:04 on dist-test-slave-8w9v"
server_key: "8147ead99a57e1e2ecaf65d00568e81d"
server_key_iv: "ee5a28d8b2750192aeb8a65d7126af75"
server_key_version: "encryptionkey@0"
I20260810 04:11:04.371506 24421 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786334794676589-31872-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786334794676589-31872-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786334794676589-31872-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:11:04.375017 24414 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.32.61" port: 36501 } attempt: 2
I20260810 04:11:04.375452 24416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:11:04.375287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.61:36501 (local address 127.0.0.1:41236)
0810 04:11:04.375374 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0810 04:11:04.375416 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.61:36501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260810 04:11:04.375805 24416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:11:04.375723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.61:36501 (local address 127.0.0.1:41248)
0810 04:11:04.375766 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0810 04:11:04.375789 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.61:36501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260810 04:11:04.375932 24414 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.32.61:36501: Network error: Client connection negotiation failed: client connection to 127.31.32.61:36501: connect: Connection refused (error 111)
I20260810 04:11:04.386371 24421 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:11:04.388276 24421 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:11:04.388397 24421 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:11:04.392477 24421 rpc_server.cc:307] RPC server started. Bound to: 127.31.32.61:36501
I20260810 04:11:04.392544 24498 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.32.61:36501 every 8 connection(s)
I20260810 04:11:04.392900 24421 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786334794676589-31872-0/minicluster-data/master-1/data/info.pb
I20260810 04:11:04.393903 24499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:11:04.395561 24499 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.32.62" port: 45203 } has no permanent_uuid. Determining permanent_uuid...
I20260810 04:11:04.395768 31872 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskws89Io/build/release/bin/kudu as pid 24421
I20260810 04:11:04.395881 31872 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786334794676589-31872-0/minicluster-data/master-1/wal/instance
I20260810 04:11:04.396113 31872 external_mini_cluster.cc:1428] Setting key ab6dc0f3b07dcbc8c6854ffa2f42c237
Aug 10 04:11:04 dist-test-slave-8w9v krb5kdc[24318](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335064, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.61@KRBTEST.COM for kudu/127.31.32.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.31.32.60@KRBTEST.COM; defaulting to no policy
I20260810 04:11:04.405421 24417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:11:04.396414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55478 (local address 127.31.32.62:45203)
0810 04:11:04.396488 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0810 04:11:04.396490 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:11:04.396676 (+ 186us) server_negotiation.cc:407] Connection header received
0810 04:11:04.397109 (+ 433us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:11:04.397111 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:11:04.397126 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:11:04.397203 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:11:04.397822 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:11:04.398328 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:11:04.399585 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:11:04.399944 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:11:04.400023 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:11:04.402256 (+ 2233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:11:04.402258 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:11:04.402258 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:11:04.402268 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:11:04.403920 (+ 1652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:11:04.404381 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:11:04.404383 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:11:04.404383 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:11:04.404450 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:11:04.404696 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:11:04.404697 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:11:04.404698 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:11:04.404851 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:11:04.404924 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0810 04:11:04.405197 (+ 273us) server_negotiation.cc:299] Negotiation successful
0810 04:11:04.405306 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 04:11:04.405560 24501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:11:04.396388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:45203 (local address 127.0.0.1:55478)
0810 04:11:04.396544 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0810 04:11:04.396584 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0810 04:11:04.397015 (+ 431us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:11:04.397250 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:11:04.397253 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:11:04.397492 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:11:04.397710 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:11:04.397716 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:11:04.398422 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:11:04.398423 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:11:04.399485 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:11:04.399488 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:11:04.400045 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:11:04.400046 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:11:04.400204 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:11:04.400797 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:11:04.400805 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:11:04.402159 (+ 1354us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:11:04.404009 (+ 1850us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:11:04.404015 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:11:04.404017 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:11:04.404283 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:11:04.404529 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:11:04.404529 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:11:04.404529 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:11:04.404626 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:11:04.404932 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:11:04.404938 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:11:04.405058 (+ 120us) client_negotiation.cc:769] Sending connection context
0810 04:11:04.405170 (+ 112us) client_negotiation.cc:240] Negotiation successful
0810 04:11:04.405336 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":86,"threads_started":1}
I20260810 04:11:04.406124 24499 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.32.61" port: 36501 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.32.60@KRBTEST.COM" created.
Aug 10 04:11:04 dist-test-slave-8w9v krb5kdc[24318](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335064, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.61@KRBTEST.COM for kudu/127.31.32.61@KRBTEST.COM
I20260810 04:11:04.414134 24501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:11:04.406395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.61:36501 (local address 127.0.0.1:41264)
0810 04:11:04.406478 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0810 04:11:04.406487 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 04:11:04.406524 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:11:04.406739 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:11:04.406739 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:11:04.406856 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:11:04.406901 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:11:04.406903 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:11:04.407797 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:11:04.407798 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:11:04.408467 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:11:04.408469 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:11:04.408911 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:11:04.408912 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:11:04.408992 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:11:04.409351 (+ 359us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:11:04.409356 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:11:04.410628 (+ 1272us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:11:04.412988 (+ 2360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:11:04.412989 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:11:04.412990 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:11:04.413177 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:11:04.413423 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:11:04.413424 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:11:04.413424 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:11:04.413479 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:11:04.413892 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:11:04.413893 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:11:04.413930 (+ 37us) client_negotiation.cc:769] Sending connection context
0810 04:11:04.413992 (+ 62us) client_negotiation.cc:240] Negotiation successful
0810 04:11:04.414068 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68}
I20260810 04:11:04.414291 24503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:11:04.406451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41264 (local address 127.31.32.61:36501)
0810 04:11:04.406602 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0810 04:11:04.406606 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 04:11:04.406624 (+ 18us) server_negotiation.cc:407] Connection header received
0810 04:11:04.406652 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:11:04.406653 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:11:04.406663 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:11:04.406724 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:11:04.406963 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:11:04.407711 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:11:04.408548 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:11:04.408835 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:11:04.408891 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:11:04.410711 (+ 1820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:11:04.410719 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:11:04.410720 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:11:04.410727 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:11:04.412907 (+ 2180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:11:04.413269 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:11:04.413273 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:11:04.413274 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:11:04.413350 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:11:04.413591 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:11:04.413592 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:11:04.413592 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:11:04.413798 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:11:04.413897 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0810 04:11:04.414052 (+ 155us) server_negotiation.cc:299] Negotiation successful
0810 04:11:04.414143 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":94,"threads_started":1}
I20260810 04:11:04.414964 24499 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.32.60" port: 45587 } has no permanent_uuid. Determining permanent_uuid...
I20260810 04:11:04.415314 24501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:11:04.415203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.60:45587 (local address 127.0.0.1:47330)
0810 04:11:04.415249 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0810 04:11:04.415292 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.60:45587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260810 04:11:04.415477 24440 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.32.60:45587: connect: Connection refused (error 111)
I20260810 04:11:04.415940 24501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:11:04.415820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.60:45587 (local address 127.0.0.1:47332)
0810 04:11:04.415900 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0810 04:11:04.415924 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.60:45587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260810 04:11:04.416090 24499 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.32.60:45587: Network error: Client connection negotiation failed: client connection to 127.31.32.60:45587: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.32.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.60.keytab.
Entry for principal kudu/127.31.32.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.60.keytab.
I20260810 04:11:04.425155 31872 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskws89Io/build/release/bin/kudu
/tmp/dist-test-taskws89Io/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786334794676589-31872-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786334794676589-31872-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786334794676589-31872-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786334794676589-31872-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.32.60:45587
--webserver_interface=127.31.32.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.60.keytab
--principal=kudu/127.31.32.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.32.20:43081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.32.62:45203,127.31.32.61:36501,127.31.32.60:45587
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786334794676589-31872-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786334794676589-31872-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786334794676589-31872-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 04:11:04.465384 24499 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.32.60" port: 45587 } attempt: 1
I20260810 04:11:04.465889 24501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:11:04.465746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.60:45587 (local address 127.0.0.1:47344)
0810 04:11:04.465814 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0810 04:11:04.465857 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.60:45587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260810 04:11:04.466243 24501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:11:04.466146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.60:45587 (local address 127.0.0.1:47356)
0810 04:11:04.466193 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0810 04:11:04.466219 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.60:45587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260810 04:11:04.466341 24499 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.32.60:45587: Network error: Client connection negotiation failed: client connection to 127.31.32.60:45587: connect: Connection refused (error 111)
I20260810 04:11:04.467067 24414 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.32.61" port: 36501 } attempt: 3
Aug 10 04:11:04 dist-test-slave-8w9v krb5kdc[24318](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335064, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.62@KRBTEST.COM for kudu/127.31.32.61@KRBTEST.COM
I20260810 04:11:04.474903 24503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:11:04.467427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41270 (local address 127.31.32.61:36501)
0810 04:11:04.467506 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0810 04:11:04.467508 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:11:04.467519 (+ 11us) server_negotiation.cc:407] Connection header received
0810 04:11:04.467569 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:11:04.467570 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:11:04.467580 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:11:04.467674 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:11:04.468018 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:11:04.468579 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:11:04.469290 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:11:04.469569 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:11:04.469660 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:11:04.471799 (+ 2139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:11:04.471801 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:11:04.471801 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:11:04.471809 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:11:04.473589 (+ 1780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:11:04.474005 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:11:04.474006 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:11:04.474007 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:11:04.474070 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:11:04.474296 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:11:04.474297 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:11:04.474298 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:11:04.474558 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:11:04.474637 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0810 04:11:04.474770 (+ 133us) server_negotiation.cc:299] Negotiation successful
0810 04:11:04.474843 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260810 04:11:04.474974 24416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:11:04.467365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.61:36501 (local address 127.0.0.1:41270)
0810 04:11:04.467416 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0810 04:11:04.467427 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 04:11:04.467508 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:11:04.467685 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:11:04.467686 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:11:04.467867 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:11:04.467935 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:11:04.467937 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:11:04.468665 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:11:04.468666 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:11:04.469205 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:11:04.469207 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:11:04.469656 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:11:04.469657 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:11:04.469741 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:11:04.470202 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:11:04.470208 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:11:04.471592 (+ 1384us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:11:04.473681 (+ 2089us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:11:04.473682 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:11:04.473683 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:11:04.473918 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:11:04.474148 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:11:04.474149 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:11:04.474149 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:11:04.474216 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:11:04.474620 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:11:04.474622 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:11:04.474676 (+ 54us) client_negotiation.cc:769] Sending connection context
0810 04:11:04.474750 (+ 74us) client_negotiation.cc:240] Negotiation successful
0810 04:11:04.474854 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260810 04:11:04.475368 24414 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.32.60" port: 45587 } has no permanent_uuid. Determining permanent_uuid...
I20260810 04:11:04.475713 24416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:11:04.475618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.60:45587 (local address 127.0.0.1:47368)
0810 04:11:04.475662 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0810 04:11:04.475686 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.60:45587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260810 04:11:04.476011 24416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:11:04.475933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.60:45587 (local address 127.0.0.1:47378)
0810 04:11:04.475962 (+ 29us) negotiation.cc:107] Waiting for socket to connect
0810 04:11:04.475981 (+ 19us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.60:45587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
W20260810 04:11:04.476087 24414 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.32.60:45587: Network error: Client connection negotiation failed: client connection to 127.31.32.60:45587: connect: Connection refused (error 111)
I20260810 04:11:04.507278 24414 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.32.60" port: 45587 } attempt: 1
I20260810 04:11:04.507882 24416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:11:04.507711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.60:45587 (local address 127.0.0.1:47388)
0810 04:11:04.507792 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0810 04:11:04.507841 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.60:45587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260810 04:11:04.508343 24416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:11:04.508212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.60:45587 (local address 127.0.0.1:47400)
0810 04:11:04.508263 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0810 04:11:04.508298 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.60:45587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260810 04:11:04.508445 24414 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.32.60:45587: Network error: Client connection negotiation failed: client connection to 127.31.32.60:45587: connect: Connection refused (error 111)
W20260810 04:11:04.510180 24507 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 04:11:04.510368 24507 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:11:04.510423 24507 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:11:04.510455 24507 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:11:04.511727 24507 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:11:04.511766 24507 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:11:04.511780 24507 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:11:04.511817 24507 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:11:04.511852 24507 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:11:04.513470 24507 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.32.20:43081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786334794676589-31872-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786334794676589-31872-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.31.32.62:45203,127.31.32.61:36501,127.31.32.60:45587
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786334794676589-31872-0/krb5kdc/kudu_127.31.32.60.keytab
--principal=kudu/127.31.32.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.32.60:45587
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786334794676589-31872-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.31.32.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24507
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786334794676589-31872-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:11:04.513725 24507 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:11:04.513962 24507 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:11:04.518910 24516 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:11:04.518882 24512 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:11:04.518882 24514 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:11:04.518999 24507 server_base.cc:1034] running on GCE node
I20260810 04:11:04.519389 24507 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:11:04.519698 24507 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:11:04.520848 24507 hybrid_clock.cc:648] HybridClock initialized: now 1786335064520816 us; error 45 us; skew 500 ppm
Aug 10 04:11:04 dist-test-slave-8w9v krb5kdc[24318](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335064, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:11:04.523483 24507 init.cc:377] Logged in from keytab as kudu/127.31.32.60@KRBTEST.COM (short username kudu)
I20260810 04:11:04.523962 24507 webserver.cc:460] Webserver started at http://127.31.32.60:33853/ using document root <none> and password file <none>
I20260810 04:11:04.524262 24507 fs_manager.cc:359] Metadata directory not provided
I20260810 04:11:04.524312 24507 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:11:04.524459 24507 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:11:04.525552 24507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786334794676589-31872-0/minicluster-data/master-2/data/instance:
uuid: "b57e07b57e564b038bdf42cd0fdfbf01"
format_stamp: "Formatted at 2026-08-10 04:11:04 on dist-test-slave-8w9v"
server_key: "b934910cba340864dbfaa3e1dce2301c"
server_key_iv: "dcf605e3826ea581ff45cecd4e682eb2"
server_key_version: "encryptionkey@0"
I20260810 04:11:04.525926 24507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786334794676589-31872-0/minicluster-data/master-2/wal/instance:
uuid: "b57e07b57e564b038bdf42cd0fdfbf01"
format_stamp: "Formatted at 2026-08-10 04:11:04 on dist-test-slave-8w9v"
server_key: "b934910cba340864dbfaa3e1dce2301c"
server_key_iv: "dcf605e3826ea581ff45cecd4e682eb2"
server_key_version: "encryptionkey@0"
I20260810 04:11:04.527281 24507 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 04:11:04.528056 24523 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:11:04.528236 24507 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 04:11:04.528352 24507 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786334794676589-31872-0/minicluster-data/master-2/data,/tmp/dist-test-taskws89Io/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786334794676589-31872-0/minicluster-data/master-2/wal
uuid: "b57e07b57e564b038bdf42cd0fdfbf01"
format_stamp: "Formatted at 2026-08-10 04:11:04 on dist-test-slave-8w9v"
server_key: "b934910cba340864dbfaa3e1dce2301c"
server_key_iv: "dcf605e3826ea581ff45cecd4e682eb2"
server_key_version: "encryptionkey@0"
I20260810 04:11:04.528409 24507 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786334794676589-31872-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786334794676589-31872-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786334794676589-31872-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:11:04.534479 24499 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.32.60" port: 45587 } attempt: 2
I20260810 04:11:04.534904 24501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:11:04.534773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.60:45587 (local address 127.0.0.1:47408)
0810 04:11:04.534831 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0810 04:11:04.534872 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.60:45587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260810 04:11:04.535256 24501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:11:04.535171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.60:45587 (local address 127.0.0.1:47420)
0810 04:11:04.535218 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0810 04:11:04.535240 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.32.60:45587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260810 04:11:04.535369 24499 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.32.60:45587: Network error: Client connection negotiation failed: client connection to 127.31.32.60:45587: connect: Connection refused (error 111)
I20260810 04:11:04.557090 24507 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:11:04.559310 24507 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:11:04.559468 24507 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:11:04.563522 24507 rpc_server.cc:307] RPC server started. Bound to: 127.31.32.60:45587
I20260810 04:11:04.563578 24580 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.32.60:45587 every 8 connection(s)
I20260810 04:11:04.563944 24507 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786334794676589-31872-0/minicluster-data/master-2/data/info.pb
I20260810 04:11:04.564967 24581 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:11:04.566466 24581 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.32.62" port: 45203 } has no permanent_uuid. Determining permanent_uuid...
I20260810 04:11:04.570797 31872 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskws89Io/build/release/bin/kudu as pid 24507
I20260810 04:11:04.570945 31872 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskws89Io/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786334794676589-31872-0/minicluster-data/master-2/wal/instance
I20260810 04:11:04.571159 31872 external_mini_cluster.cc:1428] Setting key 931ebb26901e224ef1d089cbf6c81a36
Aug 10 04:11:04 dist-test-slave-8w9v krb5kdc[24318](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335064, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.60@KRBTEST.COM for kudu/127.31.32.62@KRBTEST.COM
I20260810 04:11:04.576309 24417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:11:04.567189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55488 (local address 127.31.32.62:45203)
0810 04:11:04.567282 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0810 04:11:04.567284 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:11:04.567491 (+ 207us) server_negotiation.cc:407] Connection header received
0810 04:11:04.567995 (+ 504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:11:04.567997 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:11:04.568009 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:11:04.568061 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:11:04.568744 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:11:04.569221 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:11:04.570212 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:11:04.570425 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:11:04.570502 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:11:04.573050 (+ 2548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:11:04.573052 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:11:04.573052 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:11:04.573059 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:11:04.574609 (+ 1550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:11:04.575090 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:11:04.575091 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:11:04.575091 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:11:04.575142 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:11:04.575461 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:11:04.575462 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:11:04.575462 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:11:04.575640 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:11:04.575715 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0810 04:11:04.576105 (+ 390us) server_negotiation.cc:299] Negotiation successful
0810 04:11:04.576234 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260810 04:11:04.576511 24583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:11:04.567172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.62:45203 (local address 127.0.0.1:55488)
0810 04:11:04.567349 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0810 04:11:04.567391 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 04:11:04.567881 (+ 490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:11:04.568178 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:11:04.568183 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:11:04.568441 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:11:04.568634 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:11:04.568639 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:11:04.569315 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:11:04.569316 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:11:04.570110 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:11:04.570112 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:11:04.570521 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:11:04.570523 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:11:04.570649 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:11:04.571233 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:11:04.571244 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:11:04.572957 (+ 1713us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:11:04.574679 (+ 1722us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:11:04.574711 (+ 32us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:11:04.574713 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:11:04.575009 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:11:04.575208 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:11:04.575208 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:11:04.575209 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:11:04.575356 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:11:04.575732 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:11:04.575738 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:11:04.575925 (+ 187us) client_negotiation.cc:769] Sending connection context
0810 04:11:04.576064 (+ 139us) client_negotiation.cc:240] Negotiation successful
0810 04:11:04.576245 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":102,"threads_started":1}
I20260810 04:11:04.577114 24581 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.32.61" port: 36501 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 04:11:04 dist-test-slave-8w9v krb5kdc[24318](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335064, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.32.60@KRBTEST.COM
I20260810 04:11:04.580785 24585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:11:04.571561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47432 (local address 127.31.32.60:45587)
0810 04:11:04.571722 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0810 04:11:04.571726 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 04:11:04.571778 (+ 52us) server_negotiation.cc:407] Connection header received
0810 04:11:04.571851 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:11:04.571853 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:11:04.571869 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:11:04.571965 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:11:04.572665 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:11:04.573615 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:11:04.574728 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:11:04.575259 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:11:04.575330 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:11:04.578015 (+ 2685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:11:04.578023 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:11:04.578023 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:11:04.578029 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:11:04.579544 (+ 1515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:11:04.579964 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:11:04.579967 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:11:04.579967 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:11:04.580019 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:11:04.580239 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:11:04.580239 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:11:04.580239 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:11:04.580364 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:11:04.580541 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0810 04:11:04.580594 (+ 53us) server_negotiation.cc:299] Negotiation successful
0810 04:11:04.580665 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":69,"threads_started":1}
Aug 10 04:11:04 dist-test-slave-8w9v krb5kdc[24318](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335064, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.60@KRBTEST.COM for kudu/127.31.32.61@KRBTEST.COM
I20260810 04:11:04.583611 24414 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.32.60" port: 45587 } attempt: 2
I20260810 04:11:04.583952 24503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:11:04.577487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41282 (local address 127.31.32.61:36501)
0810 04:11:04.577564 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0810 04:11:04.577566 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:11:04.577575 (+ 9us) server_negotiation.cc:407] Connection header received
0810 04:11:04.577599 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:11:04.577600 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:11:04.577609 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:11:04.577657 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:11:04.577931 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:11:04.578454 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:11:04.579085 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:11:04.579310 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:11:04.579383 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:11:04.581192 (+ 1809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:11:04.581194 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:11:04.581194 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:11:04.581201 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:11:04.582786 (+ 1585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:11:04.583092 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:11:04.583093 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:11:04.583093 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:11:04.583175 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:11:04.583392 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:11:04.583393 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:11:04.583393 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:11:04.583507 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:11:04.583597 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0810 04:11:04.583770 (+ 173us) server_negotiation.cc:299] Negotiation successful
0810 04:11:04.583850 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 04:11:04.584097 24583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:11:04.577421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.61:36501 (local address 127.0.0.1:41282)
0810 04:11:04.577469 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0810 04:11:04.577480 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 04:11:04.577532 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:11:04.577698 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:11:04.577699 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:11:04.577811 (+ 112us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:11:04.577853 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:11:04.577854 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:11:04.578534 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:11:04.578536 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:11:04.579002 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:11:04.579004 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:11:04.579394 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:11:04.579395 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:11:04.579463 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:11:04.579811 (+ 348us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:11:04.579816 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:11:04.581104 (+ 1288us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:11:04.582866 (+ 1762us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:11:04.582867 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:11:04.582867 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:11:04.583028 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:11:04.583259 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:11:04.583259 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:11:04.583259 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:11:04.583309 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:11:04.583599 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:11:04.583599 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:11:04.583639 (+ 40us) client_negotiation.cc:769] Sending connection context
0810 04:11:04.583743 (+ 104us) client_negotiation.cc:240] Negotiation successful
0810 04:11:04.583830 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260810 04:11:04.584510 24581 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.32.60" port: 45587 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 04:11:04 dist-test-slave-8w9v krb5kdc[24318](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335064, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.32.62@KRBTEST.COM
Aug 10 04:11:04 dist-test-slave-8w9v krb5kdc[24318](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335064, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.62@KRBTEST.COM for kudu/127.31.32.60@KRBTEST.COM
I20260810 04:11:04.590030 24417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:11:04.581642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55496 (local address 127.31.32.62:45203)
0810 04:11:04.581682 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0810 04:11:04.581684 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:11:04.581752 (+ 68us) server_negotiation.cc:407] Connection header received
0810 04:11:04.581805 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:11:04.581806 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:11:04.581816 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:11:04.581886 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:11:04.582445 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:11:04.583113 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:11:04.583930 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:11:04.584267 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:11:04.584331 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:11:04.587290 (+ 2959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:11:04.587292 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:11:04.587292 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:11:04.587298 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:11:04.588850 (+ 1552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:11:04.589221 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:11:04.589222 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:11:04.589222 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:11:04.589281 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:11:04.589498 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:11:04.589499 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:11:04.589500 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:11:04.589611 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:11:04.589780 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0810 04:11:04.589823 (+ 43us) server_negotiation.cc:299] Negotiation successful
0810 04:11:04.589915 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
Aug 10 04:11:04 dist-test-slave-8w9v krb5kdc[24318](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335064, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.60@KRBTEST.COM for kudu/127.31.32.60@KRBTEST.COM
I20260810 04:11:04.591866 24416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:11:04.583848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.60:45587 (local address 127.0.0.1:47436)
0810 04:11:04.583889 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0810 04:11:04.583900 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 04:11:04.583939 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:11:04.584396 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:11:04.584397 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:11:04.584551 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:11:04.584615 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:11:04.584617 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:11:04.585499 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:11:04.585500 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:11:04.586081 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:11:04.586083 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:11:04.586600 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:11:04.586601 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:11:04.586659 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:11:04.587044 (+ 385us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:11:04.587050 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:11:04.588533 (+ 1483us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:11:04.590826 (+ 2293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:11:04.590827 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:11:04.590828 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:11:04.590986 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:11:04.591198 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:11:04.591199 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:11:04.591199 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:11:04.591255 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:11:04.591563 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:11:04.591564 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:11:04.591621 (+ 57us) client_negotiation.cc:769] Sending connection context
0810 04:11:04.591669 (+ 48us) client_negotiation.cc:240] Negotiation successful
0810 04:11:04.591763 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23}
I20260810 04:11:04.592212 24585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:11:04.584014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47436 (local address 127.31.32.60:45587)
0810 04:11:04.584247 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0810 04:11:04.584249 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:11:04.584255 (+ 6us) server_negotiation.cc:407] Connection header received
0810 04:11:04.584275 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:11:04.584276 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:11:04.584285 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:11:04.584382 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:11:04.584687 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:11:04.585421 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:11:04.586178 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:11:04.586500 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:11:04.586808 (+ 308us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:11:04.588611 (+ 1803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:11:04.588612 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:11:04.588612 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:11:04.588617 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:11:04.590631 (+ 2014us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:11:04.591060 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:11:04.591061 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:11:04.591062 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:11:04.591114 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:11:04.591343 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:11:04.591344 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:11:04.591345 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:11:04.591462 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:11:04.591975 (+ 513us) server_negotiation.cc:1035] Waiting for connection context
0810 04:11:04.592020 (+ 45us) server_negotiation.cc:299] Negotiation successful
0810 04:11:04.592084 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":217}
I20260810 04:11:04.594033 24414 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7834a3a484b641c88cd396050dc833a9: Bootstrap starting.
I20260810 04:11:04.594857 24414 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7834a3a484b641c88cd396050dc833a9: Neither blocks nor log segments found. Creating new log.
I20260810 04:11:04.594945 24583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:11:04.584728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.60:45587 (local address 127.0.0.1:47450)
0810 04:11:04.585907 (+ 1179us) negotiation.cc:107] Waiting for socket to connect
0810 04:11:04.585916 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 04:11:04.585978 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:11:04.586106 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:11:04.586107 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:11:04.586238 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:11:04.586281 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:11:04.586282 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:11:04.587055 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:11:04.587056 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:11:04.587759 (+ 703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:11:04.587762 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:11:04.588940 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:11:04.588941 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:11:04.589001 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:11:04.589384 (+ 383us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:11:04.589389 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:11:04.591444 (+ 2055us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:11:04.593673 (+ 2229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:11:04.593673 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:11:04.593674 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:11:04.593909 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:11:04.594308 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:11:04.594308 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:11:04.594309 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:11:04.594371 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:11:04.594642 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:11:04.594642 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:11:04.594680 (+ 38us) client_negotiation.cc:769] Sending connection context
0810 04:11:04.594748 (+ 68us) client_negotiation.cc:240] Negotiation successful
0810 04:11:04.594845 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1165}
I20260810 04:11:04.595063 24586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:11:04.584757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47450 (local address 127.31.32.60:45587)
0810 04:11:04.584951 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0810 04:11:04.584953 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:11:04.585967 (+ 1014us) server_negotiation.cc:407] Connection header received
0810 04:11:04.586023 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:11:04.586023 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:11:04.586032 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:11:04.586103 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:11:04.586360 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:11:04.586969 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:11:04.587851 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:11:04.588174 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:11:04.588295 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:11:04.591543 (+ 3248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:11:04.591544 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:11:04.591544 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:11:04.591550 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:11:04.593592 (+ 2042us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:11:04.593983 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:11:04.593984 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:11:04.593984 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:11:04.594038 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:11:04.594444 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:11:04.594445 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:11:04.594445 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:11:04.594573 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:11:04.594651 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0810 04:11:04.594777 (+ 126us) server_negotiation.cc:299] Negotiation successful
0810 04:11:04.594919 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":52,"server-negotiator.queue_time_us":169,"thread_start_us":94,"threads_started":1}
I20260810 04:11:04.595225 24414 log.cc:824] T 00000000000000000000000000000000 P 7834a3a484b641c88cd396050dc833a9: Log is configured to *not* fsync() on all Append() calls
I20260810 04:11:04.596098 24414 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7834a3a484b641c88cd396050dc833a9: No bootstrap required, opened a new log
Aug 10 04:11:04 dist-test-slave-8w9v krb5kdc[24318](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335064, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.32.61@KRBTEST.COM
I20260810 04:11:04.596513 24581 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b57e07b57e564b038bdf42cd0fdfbf01: Bootstrap starting.
I20260810 04:11:04.597453 24581 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b57e07b57e564b038bdf42cd0fdfbf01: Neither blocks nor log segments found. Creating new log.
I20260810 04:11:04.598030 24581 log.cc:824] T 00000000000000000000000000000000 P b57e07b57e564b038bdf42cd0fdfbf01: Log is configured to *not* fsync() on all Append() calls
I20260810 04:11:04.598457 24414 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7834a3a484b641c88cd396050dc833a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7834a3a484b641c88cd396050dc833a9" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 45203 } } peers { permanent_uuid: "50cfd406d47a44d9a1ac795c5e11b88c" member_type: VOTER last_known_addr { host: "127.31.32.61" port: 36501 } } peers { permanent_uuid: "b57e07b57e564b038bdf42cd0fdfbf01" member_type: VOTER last_known_addr { host: "127.31.32.60" port: 45587 } }
I20260810 04:11:04.598600 24414 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7834a3a484b641c88cd396050dc833a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7834a3a484b641c88cd396050dc833a9, State: Initialized, Role: FOLLOWER
I20260810 04:11:04.598690 24414 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7834a3a484b641c88cd396050dc833a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7834a3a484b641c88cd396050dc833a9" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 45203 } } peers { permanent_uuid: "50cfd406d47a44d9a1ac795c5e11b88c" member_type: VOTER last_known_addr { host: "127.31.32.61" port: 36501 } } peers { permanent_uuid: "b57e07b57e564b038bdf42cd0fdfbf01" member_type: VOTER last_known_addr { host: "127.31.32.60" port: 45587 } }
I20260810 04:11:04.598893 24581 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b57e07b57e564b038bdf42cd0fdfbf01: No bootstrap required, opened a new log
I20260810 04:11:04.599045 24414 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7834a3a484b641c88cd396050dc833a9 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:11:04.599025 24590 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7834a3a484b641c88cd396050dc833a9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7834a3a484b641c88cd396050dc833a9" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 45203 } } peers { permanent_uuid: "50cfd406d47a44d9a1ac795c5e11b88c" member_type: VOTER last_known_addr { host: "127.31.32.61" port: 36501 } } peers { permanent_uuid: "b57e07b57e564b038bdf42cd0fdfbf01" member_type: VOTER last_known_addr { host: "127.31.32.60" port: 45587 } } }
I20260810 04:11:04.599225 24590 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7834a3a484b641c88cd396050dc833a9 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 04:11:04.599722 24503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:11:04.591107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41284 (local address 127.31.32.61:36501)
0810 04:11:04.591154 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0810 04:11:04.591156 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:11:04.591205 (+ 49us) server_negotiation.cc:407] Connection header received
0810 04:11:04.591288 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:11:04.591289 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:11:04.591300 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:11:04.591355 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:11:04.592192 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:11:04.592845 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:11:04.593535 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:11:04.593837 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:11:04.593910 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:11:04.596901 (+ 2991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:11:04.596902 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:11:04.596902 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:11:04.596909 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:11:04.598399 (+ 1490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:11:04.598752 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:11:04.598753 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:11:04.598754 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:11:04.598810 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:11:04.599011 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:11:04.599012 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:11:04.599012 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:11:04.599136 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:11:04.599342 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0810 04:11:04.599554 (+ 212us) server_negotiation.cc:299] Negotiation successful
0810 04:11:04.599622 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
W20260810 04:11:04.601231 24602 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7834a3a484b641c88cd396050dc833a9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 04:11:04.601295 24602 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 04:11:04.601370 31872 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 04:11:04.602003 24581 raft_consensus.cc:348] T 00000000000000000000000000000000 P b57e07b57e564b038bdf42cd0fdfbf01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7834a3a484b641c88cd396050dc833a9" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 45203 } } peers { permanent_uuid: "50cfd406d47a44d9a1ac795c5e11b88c" member_type: VOTER last_known_addr { host: "127.31.32.61" port: 36501 } } peers { permanent_uuid: "b57e07b57e564b038bdf42cd0fdfbf01" member_type: VOTER last_known_addr { host: "127.31.32.60" port: 45587 } }
I20260810 04:11:04.602137 24581 raft_consensus.cc:729] T 00000000000000000000000000000000 P b57e07b57e564b038bdf42cd0fdfbf01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b57e07b57e564b038bdf42cd0fdfbf01, State: Initialized, Role: FOLLOWER
I20260810 04:11:04.602217 24581 consensus_queue.cc:260] T 00000000000000000000000000000000 P b57e07b57e564b038bdf42cd0fdfbf01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7834a3a484b641c88cd396050dc833a9" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 45203 } } peers { permanent_uuid: "50cfd406d47a44d9a1ac795c5e11b88c" member_type: VOTER last_known_addr { host: "127.31.32.61" port: 36501 } } peers { permanent_uuid: "b57e07b57e564b038bdf42cd0fdfbf01" member_type: VOTER last_known_addr { host: "127.31.32.60" port: 45587 } }
I20260810 04:11:04.602312 24383 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "7834a3a484b641c88cd396050dc833a9"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55496
I20260810 04:11:04.602417 24383 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7834a3a484b641c88cd396050dc833a9 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260810 04:11:04.602463 24383 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7834a3a484b641c88cd396050dc833a9 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:11:04.602509 24581 sys_catalog.cc:564] T 00000000000000000000000000000000 P b57e07b57e564b038bdf42cd0fdfbf01 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:11:04.603734 24383 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7834a3a484b641c88cd396050dc833a9 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7834a3a484b641c88cd396050dc833a9" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 45203 } } peers { permanent_uuid: "50cfd406d47a44d9a1ac795c5e11b88c" member_type: VOTER last_known_addr { host: "127.31.32.61" port: 36501 } } peers { permanent_uuid: "b57e07b57e564b038bdf42cd0fdfbf01" member_type: VOTER last_known_addr { host: "127.31.32.60" port: 45587 } }
I20260810 04:11:04.603976 24383 leader_election.cc:288] T 00000000000000000000000000000000 P 7834a3a484b641c88cd396050dc833a9 [CANDIDATE]: Term 1 election: Requested vote from peers 50cfd406d47a44d9a1ac795c5e11b88c (127.31.32.61:36501), b57e07b57e564b038bdf42cd0fdfbf01 (127.31.32.60:45587)
I20260810 04:11:04.601943 24334 master_runner.cc:186] Error getting master registration for 127.31.32.61:36501: OK, instance_id { permanent_uuid: "50cfd406d47a44d9a1ac795c5e11b88c" instance_seqno: 1786335064391279 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 04:11:04.604382 24334 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 04:11:04.603865 24604 sys_catalog.cc:455] T 00000000000000000000000000000000 P b57e07b57e564b038bdf42cd0fdfbf01 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7834a3a484b641c88cd396050dc833a9" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 45203 } } peers { permanent_uuid: "50cfd406d47a44d9a1ac795c5e11b88c" member_type: VOTER last_known_addr { host: "127.31.32.61" port: 36501 } } peers { permanent_uuid: "b57e07b57e564b038bdf42cd0fdfbf01" member_type: VOTER last_known_addr { host: "127.31.32.60" port: 45587 } } }
I20260810 04:11:04.604579 24552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7834a3a484b641c88cd396050dc833a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "b57e07b57e564b038bdf42cd0fdfbf01"
I20260810 04:11:04.605717 24552 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b57e07b57e564b038bdf42cd0fdfbf01 [term 0 FOLLOWER]: Advancing to term 1
W20260810 04:11:04.604841 24616 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b57e07b57e564b038bdf42cd0fdfbf01: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 04:11:04.606033 24616 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 04:11:04.606657 24552 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b57e07b57e564b038bdf42cd0fdfbf01 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7834a3a484b641c88cd396050dc833a9 in term 1.
I20260810 04:11:04.606812 24604 sys_catalog.cc:458] T 00000000000000000000000000000000 P b57e07b57e564b038bdf42cd0fdfbf01 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 04:11:04.606854 24355 leader_election.cc:302] T 00000000000000000000000000000000 P 7834a3a484b641c88cd396050dc833a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7834a3a484b641c88cd396050dc833a9, b57e07b57e564b038bdf42cd0fdfbf01; no voters:
I20260810 04:11:04.607105 24590 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7834a3a484b641c88cd396050dc833a9 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:11:04.607306 24590 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7834a3a484b641c88cd396050dc833a9 [term 1 LEADER]: Becoming Leader. State: Replica: 7834a3a484b641c88cd396050dc833a9, State: Running, Role: LEADER
I20260810 04:11:04.607460 24590 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7834a3a484b641c88cd396050dc833a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7834a3a484b641c88cd396050dc833a9" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 45203 } } peers { permanent_uuid: "50cfd406d47a44d9a1ac795c5e11b88c" member_type: VOTER last_known_addr { host: "127.31.32.61" port: 36501 } } peers { permanent_uuid: "b57e07b57e564b038bdf42cd0fdfbf01" member_type: VOTER last_known_addr { host: "127.31.32.60" port: 45587 } }
I20260810 04:11:04.607986 24590 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7834a3a484b641c88cd396050dc833a9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7834a3a484b641c88cd396050dc833a9. Latest consensus state: current_term: 1 leader_uuid: "7834a3a484b641c88cd396050dc833a9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7834a3a484b641c88cd396050dc833a9" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 45203 } } peers { permanent_uuid: "50cfd406d47a44d9a1ac795c5e11b88c" member_type: VOTER last_known_addr { host: "127.31.32.61" port: 36501 } } peers { permanent_uuid: "b57e07b57e564b038bdf42cd0fdfbf01" member_type: VOTER last_known_addr { host: "127.31.32.60" port: 45587 } } }
I20260810 04:11:04.608174 24590 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7834a3a484b641c88cd396050dc833a9 [sys.catalog]: This master's current role is: LEADER
I20260810 04:11:04.607692 24507 master_runner.cc:186] Error getting master registration for 127.31.32.61:36501: OK, instance_id { permanent_uuid: "50cfd406d47a44d9a1ac795c5e11b88c" instance_seqno: 1786335064391279 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 04:11:04.609973 24507 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 04:11:04.611634 24627 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 04:11:04.611965 24627 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 04:11:04.615710 24503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:11:04.605772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41290 (local address 127.31.32.61:36501)
0810 04:11:04.605880 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0810 04:11:04.605882 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:11:04.606384 (+ 502us) server_negotiation.cc:407] Connection header received
0810 04:11:04.606477 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:11:04.606478 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:11:04.606488 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:11:04.606518 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:11:04.608944 (+ 2426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:11:04.609665 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:11:04.610816 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:11:04.611153 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:11:04.611197 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:11:04.612865 (+ 1668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:11:04.612867 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:11:04.612868 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:11:04.612875 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:11:04.614552 (+ 1677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:11:04.614861 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:11:04.614862 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:11:04.614862 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:11:04.614918 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:11:04.615123 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:11:04.615124 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:11:04.615125 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:11:04.615243 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:11:04.615309 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0810 04:11:04.615530 (+ 221us) server_negotiation.cc:299] Negotiation successful
0810 04:11:04.615609 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260810 04:11:04.623515 24417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:11:04.605955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55504 (local address 127.31.32.62:45203)
0810 04:11:04.606004 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0810 04:11:04.606006 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:11:04.610521 (+ 4515us) server_negotiation.cc:407] Connection header received
0810 04:11:04.610757 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:11:04.610758 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:11:04.610782 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:11:04.610813 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:11:04.612953 (+ 2140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:11:04.613685 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:11:04.617436 (+ 3751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:11:04.617747 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:11:04.617793 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:11:04.620552 (+ 2759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:11:04.620554 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:11:04.620555 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:11:04.620562 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:11:04.622220 (+ 1658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:11:04.622625 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:11:04.622626 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:11:04.622626 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:11:04.622678 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:11:04.622888 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:11:04.622889 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:11:04.622889 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:11:04.623024 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:11:04.623126 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0810 04:11:04.623319 (+ 193us) server_negotiation.cc:299] Negotiation successful
0810 04:11:04.623403 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260810 04:11:04.624727 24586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:11:04.607040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47462 (local address 127.31.32.60:45587)
0810 04:11:04.607077 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0810 04:11:04.607078 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 04:11:04.610457 (+ 3379us) server_negotiation.cc:407] Connection header received
0810 04:11:04.610905 (+ 448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:11:04.610906 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:11:04.610915 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:11:04.610943 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:11:04.613823 (+ 2880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:11:04.614547 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:11:04.617948 (+ 3401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:11:04.618296 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:11:04.618339 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:11:04.620906 (+ 2567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:11:04.620907 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:11:04.620907 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:11:04.620915 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:11:04.622939 (+ 2024us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:11:04.623557 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:11:04.623558 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:11:04.623558 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:11:04.623627 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:11:04.623939 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:11:04.623940 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:11:04.623940 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:11:04.624139 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:11:04.624233 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0810 04:11:04.624542 (+ 309us) server_negotiation.cc:299] Negotiation successful
0810 04:11:04.624631 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260810 04:11:04.626513 24499 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.32.60" port: 45587 } attempt: 3
I20260810 04:11:04.629223 24552 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b57e07b57e564b038bdf42cd0fdfbf01 [term 1 FOLLOWER]: Refusing update from remote peer 7834a3a484b641c88cd396050dc833a9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:11:04.630275 24625 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7834a3a484b641c88cd396050dc833a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b57e07b57e564b038bdf42cd0fdfbf01" member_type: VOTER last_known_addr { host: "127.31.32.60" port: 45587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Aug 10 04:11:04 dist-test-slave-8w9v krb5kdc[24318](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335064, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.32.61@KRBTEST.COM for kudu/127.31.32.60@KRBTEST.COM
I20260810 04:11:04.631691 24604 sys_catalog.cc:455] T 00000000000000000000000000000000 P b57e07b57e564b038bdf42cd0fdfbf01 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7834a3a484b641c88cd396050dc833a9. Latest consensus state: current_term: 1 leader_uuid: "7834a3a484b641c88cd396050dc833a9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7834a3a484b641c88cd396050dc833a9" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 45203 } } peers { permanent_uuid: "50cfd406d47a44d9a1ac795c5e11b88c" member_type: VOTER last_known_addr { host: "127.31.32.61" port: 36501 } } peers { permanent_uuid: "b57e07b57e564b038bdf42cd0fdfbf01" member_type: VOTER last_known_addr { host: "127.31.32.60" port: 45587 } } }
I20260810 04:11:04.631769 24604 sys_catalog.cc:458] T 00000000000000000000000000000000 P b57e07b57e564b038bdf42cd0fdfbf01 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 04:11:04.634554 24625 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7834a3a484b641c88cd396050dc833a9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7834a3a484b641c88cd396050dc833a9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7834a3a484b641c88cd396050dc833a9" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 45203 } } peers { permanent_uuid: "50cfd406d47a44d9a1ac795c5e11b88c" member_type: VOTER last_known_addr { host: "127.31.32.61" port: 36501 } } peers { permanent_uuid: "b57e07b57e564b038bdf42cd0fdfbf01" member_type: VOTER last_known_addr { host: "127.31.32.60" port: 45587 } } }
I20260810 04:11:04.634662 24625 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7834a3a484b641c88cd396050dc833a9 [sys.catalog]: This master's current role is: LEADER
I20260810 04:11:04.635156 24604 sys_catalog.cc:455] T 00000000000000000000000000000000 P b57e07b57e564b038bdf42cd0fdfbf01 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7834a3a484b641c88cd396050dc833a9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7834a3a484b641c88cd396050dc833a9" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 45203 } } peers { permanent_uuid: "50cfd406d47a44d9a1ac795c5e11b88c" member_type: VOTER last_known_addr { host: "127.31.32.61" port: 36501 } } peers { permanent_uuid: "b57e07b57e564b038bdf42cd0fdfbf01" member_type: VOTER last_known_addr { host: "127.31.32.60" port: 45587 } } }
I20260810 04:11:04.635242 24604 sys_catalog.cc:458] T 00000000000000000000000000000000 P b57e07b57e564b038bdf42cd0fdfbf01 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 04:11:04.635596 24630 mvcc.cc:205] Tried to move back new op lower bound from 7316828424718221312 to 7316828424633311232. Current Snapshot: MvccSnapshot[applied={T|T < 7316828424718221312}]
I20260810 04:11:04.635720 24501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:11:04.626781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.32.60:45587 (local address 127.0.0.1:47476)
0810 04:11:04.626835 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0810 04:11:04.626848 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 04:11:04.626895 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:11:04.627064 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:11:04.627064 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:11:04.627238 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:11:04.627296 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:11:04.627298 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:11:04.628271 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:11:04.628284 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:11:04.629144 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:11:04.629147 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:11:04.629862 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:11:04.629863 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:11:04.629930 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:11:04.630543 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:11:04.630548 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:11:04.632143 (+ 1595us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:11:04.634212 (+ 2069us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:11:04.634213 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:11:04.634213 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:11:04.634439 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:11:04.634836 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:11:04.634838 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:11:04.634838 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:11:04.634912 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:11:04.635416 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:11:04.635417 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:11:04.635472 (+ 55us) client_negotiation.cc:769] Sending connection context
0810 04:11:04.635528 (+ 56us) client_negotiation.cc:240] Negotiation successful
0810 04:11:04.635622 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260810 04:11:04.636590 24586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:11:04.626898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47476 (local address 127.31.32.60:45587)
0810 04:11:04.626944 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0810 04:11:04.626946 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:11:04.626954 (+ 8us) server_negotiation.cc:407] Connection header received
0810 04:11:04.626992 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:11:04.626993 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:11:04.627002 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:11:04.627060 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:11:04.627377 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:11:04.628174 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:11:04.629438 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:11:04.629739 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:11:04.629800 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:11:04.632306 (+ 2506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:11:04.632307 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:11:04.632308 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:11:04.632314 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:11:04.634104 (+ 1790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:11:04.634527 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:11:04.634529 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:11:04.634529 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:11:04.634609 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:11:04.634991 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:11:04.634992 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:11:04.634992 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:11:04.635206 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:11:04.636221 (+ 1015us) server_negotiation.cc:1035] Waiting for connection context
0810 04:11:04.636343 (+ 122us) server_negotiation.cc:299] Negotiation successful
0810 04:11:04.636465 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260810 04:11:04.637081 24627 catalog_manager.cc:1347] Generated new cluster ID: b4e07ad706b641a5b786fd685cb3197b
I20260810 04:11:04.637132 24627 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 04:11:04.638262 24499 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50cfd406d47a44d9a1ac795c5e11b88c: Bootstrap starting.
I20260810 04:11:04.638945 24499 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 50cfd406d47a44d9a1ac795c5e11b88c: Neither blocks nor log segments found. Creating new log.
I20260810 04:11:04.639222 24499 log.cc:824] T 00000000000000000000000000000000 P 50cfd406d47a44d9a1ac795c5e11b88c: Log is configured to *not* fsync() on all Append() calls
I20260810 04:11:04.644917 24499 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50cfd406d47a44d9a1ac795c5e11b88c: No bootstrap required, opened a new log
I20260810 04:11:04.646893 24499 raft_consensus.cc:348] T 00000000000000000000000000000000 P 50cfd406d47a44d9a1ac795c5e11b88c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7834a3a484b641c88cd396050dc833a9" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 45203 } } peers { permanent_uuid: "50cfd406d47a44d9a1ac795c5e11b88c" member_type: VOTER last_known_addr { host: "127.31.32.61" port: 36501 } } peers { permanent_uuid: "b57e07b57e564b038bdf42cd0fdfbf01" member_type: VOTER last_known_addr { host: "127.31.32.60" port: 45587 } }
I20260810 04:11:04.647022 24499 raft_consensus.cc:729] T 00000000000000000000000000000000 P 50cfd406d47a44d9a1ac795c5e11b88c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50cfd406d47a44d9a1ac795c5e11b88c, State: Initialized, Role: FOLLOWER
I20260810 04:11:04.647028 24627 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 04:11:04.647094 24499 consensus_queue.cc:260] T 00000000000000000000000000000000 P 50cfd406d47a44d9a1ac795c5e11b88c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7834a3a484b641c88cd396050dc833a9" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 45203 } } peers { permanent_uuid: "50cfd406d47a44d9a1ac795c5e11b88c" member_type: VOTER last_known_addr { host: "127.31.32.61" port: 36501 } } peers { permanent_uuid: "b57e07b57e564b038bdf42cd0fdfbf01" member_type: VOTER last_known_addr { host: "127.31.32.60" port: 45587 } }
I20260810 04:11:04.647401 24499 sys_catalog.cc:564] T 00000000000000000000000000000000 P 50cfd406d47a44d9a1ac795c5e11b88c [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:11:04.647679 24627 catalog_manager.cc:1504] Loading token signing keys...
I20260810 04:11:04.648211 24468 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 50cfd406d47a44d9a1ac795c5e11b88c [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:11:04.649071 24468 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 50cfd406d47a44d9a1ac795c5e11b88c [term 1 FOLLOWER]: Refusing update from remote peer 7834a3a484b641c88cd396050dc833a9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:11:04.649353 24625 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7834a3a484b641c88cd396050dc833a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50cfd406d47a44d9a1ac795c5e11b88c" member_type: VOTER last_known_addr { host: "127.31.32.61" port: 36501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:11:04.649806 24469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7834a3a484b641c88cd396050dc833a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "50cfd406d47a44d9a1ac795c5e11b88c"
I20260810 04:11:04.650614 24469 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 50cfd406d47a44d9a1ac795c5e11b88c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7834a3a484b641c88cd396050dc833a9 in term 1.
W20260810 04:11:04.653594 24653 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 50cfd406d47a44d9a1ac795c5e11b88c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 04:11:04.653792 24653 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 04:11:04.654003 24644 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50cfd406d47a44d9a1ac795c5e11b88c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7834a3a484b641c88cd396050dc833a9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7834a3a484b641c88cd396050dc833a9" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 45203 } } peers { permanent_uuid: "50cfd406d47a44d9a1ac795c5e11b88c" member_type: VOTER last_known_addr { host: "127.31.32.61" port: 36501 } } peers { permanent_uuid: "b57e07b57e564b038bdf42cd0fdfbf01" member_type: VOTER last_known_addr { host: "127.31.32.60" port: 45587 } } }
I20260810 04:11:04.654122 24644 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50cfd406d47a44d9a1ac795c5e11b88c [sys.catalog]: This master's current role is: FOLLOWER
I20260810 04:11:04.654207 24644 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50cfd406d47a44d9a1ac795c5e11b88c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7834a3a484b641c88cd396050dc833a9. Latest consensus state: current_term: 1 leader_uuid: "7834a3a484b641c88cd396050dc833a9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7834a3a484b641c88cd396050dc833a9" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 45203 } } peers { permanent_uuid: "50cfd406d47a44d9a1ac795c5e11b88c" member_type: VOTER last_known_addr { host: "127.31.32.61" port: 36501 } } peers { permanent_uuid: "b57e07b57e564b038bdf42cd0fdfbf01" member_type: VOTER last_known_addr { host: "127.31.32.60" port: 45587 } } }
I20260810 04:11:04.654310 24644 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50cfd406d47a44d9a1ac795c5e11b88c [sys.catalog]: This master's current role is: FOLLOWER
I20260810 04:11:04.654405 24644 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50cfd406d47a44d9a1ac795c5e11b88c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7834a3a484b641c88cd396050dc833a9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7834a3a484b641c88cd396050dc833a9" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 45203 } } peers { permanent_uuid: "50cfd406d47a44d9a1ac795c5e11b88c" member_type: VOTER last_known_addr { host: "127.31.32.61" port: 36501 } } peers { permanent_uuid: "b57e07b57e564b038bdf42cd0fdfbf01" member_type: VOTER last_known_addr { host: "127.31.32.60" port: 45587 } } }
I20260810 04:11:04.654493 24644 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50cfd406d47a44d9a1ac795c5e11b88c [sys.catalog]: This master's current role is: FOLLOWER
I20260810 04:11:04.654978 24645 mvcc.cc:205] Tried to move back new op lower bound from 7316828424789274624 to 7316828424633311232. Current Snapshot: MvccSnapshot[applied={T|T < 7316828424718221312}]
I20260810 04:11:04.656497 24625 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7834a3a484b641c88cd396050dc833a9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7834a3a484b641c88cd396050dc833a9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7834a3a484b641c88cd396050dc833a9" member_type: VOTER last_known_addr { host: "127.31.32.62" port: 45203 } } peers { permanent_uuid: "50cfd406d47a44d9a1ac795c5e11b88c" member_type: VOTER last_known_addr { host: "127.31.32.61" port: 36501 } } peers { permanent_uuid: "b57e07b57e564b038bdf42cd0fdfbf01" member_type: VOTER last_known_addr { host: "127.31.32.60" port: 45587 } } }
I20260810 04:11:04.656589 24625 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7834a3a484b641c88cd396050dc833a9 [sys.catalog]: This master's current role is: LEADER
I20260810 04:11:04.657802 24627 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7834a3a484b641c88cd396050dc833a9: Generated new TSK 0
I20260810 04:11:04.657977 24627 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 04:11:04.687219 24503 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:11:04.685532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41296 (local address 127.31.32.61:36501)
0810 04:11:04.685579 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0810 04:11:04.685581 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:11:04.686552 (+ 971us) server_negotiation.cc:407] Connection header received
0810 04:11:04.686655 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:11:04.686656 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:11:04.686668 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:11:04.686698 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:11:04.687147 (+ 449us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41296: BlockingRecv error: recv got EOF from 127.0.0.1:41296 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260810 04:11:04.688167 24417 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:11:04.685934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55506 (local address 127.31.32.62:45203)
0810 04:11:04.685976 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0810 04:11:04.685977 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 04:11:04.687624 (+ 1647us) server_negotiation.cc:407] Connection header received
0810 04:11:04.687721 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:11:04.687722 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:11:04.687733 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:11:04.687761 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:11:04.688071 (+ 310us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55506: BlockingRecv error: recv got EOF from 127.0.0.1:55506 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260810 04:11:04.689123 24586 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:11:04.686289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47488 (local address 127.31.32.60:45587)
0810 04:11:04.686330 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0810 04:11:04.686332 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:11:04.688506 (+ 2174us) server_negotiation.cc:407] Connection header received
0810 04:11:04.688668 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:11:04.688670 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:11:04.688679 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:11:04.688709 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:11:04.689048 (+ 339us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47488: BlockingRecv error: recv got EOF from 127.0.0.1:47488 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260810 04:11:04.691538 24503 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:11:04.690823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41310 (local address 127.31.32.61:36501)
0810 04:11:04.690869 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0810 04:11:04.690871 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:11:04.691038 (+ 167us) server_negotiation.cc:407] Connection header received
0810 04:11:04.691141 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:11:04.691142 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:11:04.691160 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:11:04.691189 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:11:04.691496 (+ 307us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41310: BlockingRecv error: recv got EOF from 127.0.0.1:41310 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260810 04:11:04.694516 24503 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:11:04.693783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41324 (local address 127.31.32.61:36501)
0810 04:11:04.693815 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0810 04:11:04.693816 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 04:11:04.694047 (+ 231us) server_negotiation.cc:407] Connection header received
0810 04:11:04.694137 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:11:04.694137 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:11:04.694146 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:11:04.694175 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:11:04.694475 (+ 300us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41324: BlockingRecv error: recv got EOF from 127.0.0.1:41324 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260810 04:11:04.698231 24503 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:11:04.697511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41334 (local address 127.31.32.61:36501)
0810 04:11:04.697544 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0810 04:11:04.697545 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 04:11:04.697770 (+ 225us) server_negotiation.cc:407] Connection header received
0810 04:11:04.697862 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:11:04.697863 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:11:04.697872 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:11:04.697900 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:11:04.698194 (+ 294us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41334: BlockingRecv error: recv got EOF from 127.0.0.1:41334 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260810 04:11:04.705163 24503 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:11:04.704420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41340 (local address 127.31.32.61:36501)
0810 04:11:04.704456 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0810 04:11:04.704458 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:11:04.704669 (+ 211us) server_negotiation.cc:407] Connection header received
0810 04:11:04.704764 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:11:04.704765 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:11:04.704775 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:11:04.704806 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:11:04.705122 (+ 316us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41340: BlockingRecv error: recv got EOF from 127.0.0.1:41340 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260810 04:11:04.715591 24503 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:11:04.714469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41356 (local address 127.31.32.61:36501)
0810 04:11:04.714511 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0810 04:11:04.714513 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:11:04.714812 (+ 299us) server_negotiation.cc:407] Connection header received
0810 04:11:04.714916 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:11:04.714917 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:11:04.714927 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:11:04.714957 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:11:04.715541 (+ 584us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41356: BlockingRecv error: recv got EOF from 127.0.0.1:41356 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260810 04:11:04.733949 24503 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:11:04.732968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41368 (local address 127.31.32.61:36501)
0810 04:11:04.733012 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0810 04:11:04.733014 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:11:04.733310 (+ 296us) server_negotiation.cc:407] Connection header received
0810 04:11:04.733424 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:11:04.733425 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:11:04.733438 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:11:04.733471 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:11:04.733865 (+ 394us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41368: BlockingRecv error: recv got EOF from 127.0.0.1:41368 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260810 04:11:04.768678 24503 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:11:04.767528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41384 (local address 127.31.32.61:36501)
0810 04:11:04.767579 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0810 04:11:04.767581 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:11:04.768001 (+ 420us) server_negotiation.cc:407] Connection header received
0810 04:11:04.768114 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:11:04.768115 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:11:04.768146 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:11:04.768180 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:11:04.768617 (+ 437us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41384: BlockingRecv error: recv got EOF from 127.0.0.1:41384 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260810 04:11:04.834971 24503 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:11:04.834281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41388 (local address 127.31.32.61:36501)
0810 04:11:04.834338 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0810 04:11:04.834340 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:11:04.834444 (+ 104us) server_negotiation.cc:407] Connection header received
0810 04:11:04.834509 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:11:04.834510 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:11:04.834523 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:11:04.834583 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:11:04.834913 (+ 330us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41388: BlockingRecv error: recv got EOF from 127.0.0.1:41388 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
W20260810 04:11:04.964979 24503 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:11:04.964179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41394 (local address 127.31.32.61:36501)
0810 04:11:04.964264 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0810 04:11:04.964266 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:11:04.964372 (+ 106us) server_negotiation.cc:407] Connection header received
0810 04:11:04.964467 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:11:04.964469 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:11:04.964481 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:11:04.964553 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:11:04.964907 (+ 354us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41394: BlockingRecv error: recv got EOF from 127.0.0.1:41394 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260810 04:11:05.223371 24503 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:11:05.222519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41406 (local address 127.31.32.61:36501)
0810 04:11:05.222594 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0810 04:11:05.222596 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:11:05.222646 (+ 50us) server_negotiation.cc:407] Connection header received
0810 04:11:05.222712 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:11:05.222713 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:11:05.222726 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:11:05.222801 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:11:05.223288 (+ 487us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41406: BlockingRecv error: recv got EOF from 127.0.0.1:41406 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
I20260810 04:11:05.606586 24616 catalog_manager.cc:1259] Loaded cluster ID: b4e07ad706b641a5b786fd685cb3197b
I20260810 04:11:05.606662 24616 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b57e07b57e564b038bdf42cd0fdfbf01: loading cluster ID for follower catalog manager: success
I20260810 04:11:05.607519 24616 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b57e07b57e564b038bdf42cd0fdfbf01: acquiring CA information for follower catalog manager: success
I20260810 04:11:05.607901 24616 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b57e07b57e564b038bdf42cd0fdfbf01: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 04:11:05.654286 24653 catalog_manager.cc:1259] Loaded cluster ID: b4e07ad706b641a5b786fd685cb3197b
I20260810 04:11:05.654390 24653 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 50cfd406d47a44d9a1ac795c5e11b88c: loading cluster ID for follower catalog manager: success
I20260810 04:11:05.655285 24653 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 50cfd406d47a44d9a1ac795c5e11b88c: acquiring CA information for follower catalog manager: success
I20260810 04:11:05.655638 24653 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 50cfd406d47a44d9a1ac795c5e11b88c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 04:11:05.739425 24730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:11:05.736595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41414 (local address 127.31.32.61:36501)
0810 04:11:05.736766 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0810 04:11:05.736769 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:11:05.736782 (+ 13us) server_negotiation.cc:407] Connection header received
0810 04:11:05.736860 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:11:05.736861 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:11:05.736930 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:11:05.737007 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 04:11:05.737151 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:11:05.737635 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:11:05.738418 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:11:05.738763 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:11:05.738851 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:11:05.739008 (+ 157us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 04:11:05.739161 (+ 153us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 04:11:05.739234 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0810 04:11:05.739284 (+ 50us) server_negotiation.cc:299] Negotiation successful
0810 04:11:05.739307 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":85,"threads_started":1}
I20260810 04:11:05.743760 24739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:11:05.741341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47502 (local address 127.31.32.60:45587)
0810 04:11:05.741473 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0810 04:11:05.741476 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:11:05.741483 (+ 7us) server_negotiation.cc:407] Connection header received
0810 04:11:05.741524 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:11:05.741524 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:11:05.741568 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:11:05.741654 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 04:11:05.741746 (+ 92us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:11:05.742231 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:11:05.742964 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:11:05.743176 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:11:05.743250 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:11:05.743410 (+ 160us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 04:11:05.743515 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 04:11:05.743595 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0810 04:11:05.743648 (+ 53us) server_negotiation.cc:299] Negotiation successful
0810 04:11:05.743670 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":62,"threads_started":1}
I20260810 04:11:05.749043 24730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:11:05.745911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41422 (local address 127.31.32.61:36501)
0810 04:11:05.745948 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0810 04:11:05.745950 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:11:05.746164 (+ 214us) server_negotiation.cc:407] Connection header received
0810 04:11:05.746245 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:11:05.746246 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:11:05.746254 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:11:05.746308 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 04:11:05.746404 (+ 96us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:11:05.747109 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:11:05.748226 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:11:05.748565 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:11:05.748622 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:11:05.748772 (+ 150us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 04:11:05.748804 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 04:11:05.748851 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0810 04:11:05.748934 (+ 83us) server_negotiation.cc:299] Negotiation successful
0810 04:11:05.748967 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260810 04:11:05.751003 24739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:11:05.747592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47516 (local address 127.31.32.60:45587)
0810 04:11:05.747647 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0810 04:11:05.747649 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:11:05.747859 (+ 210us) server_negotiation.cc:407] Connection header received
0810 04:11:05.747925 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:11:05.747926 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:11:05.747935 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:11:05.747994 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 04:11:05.748164 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:11:05.748952 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:11:05.750210 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:11:05.750494 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:11:05.750560 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:11:05.750749 (+ 189us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 04:11:05.750780 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 04:11:05.750824 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0810 04:11:05.750900 (+ 76us) server_negotiation.cc:299] Negotiation successful
0810 04:11:05.750930 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260810 04:11:05.752328 24747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:11:05.746037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55514 (local address 127.31.32.62:45203)
0810 04:11:05.746180 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0810 04:11:05.746183 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:11:05.748080 (+ 1897us) server_negotiation.cc:407] Connection header received
0810 04:11:05.748171 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:11:05.748173 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:11:05.748224 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:11:05.748305 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 04:11:05.749401 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:11:05.750078 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:11:05.751161 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:11:05.751466 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:11:05.751542 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:11:05.751853 (+ 311us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 04:11:05.751974 (+ 121us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 04:11:05.752049 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0810 04:11:05.752215 (+ 166us) server_negotiation.cc:299] Negotiation successful
0810 04:11:05.752243 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":70,"threads_started":1}
I20260810 04:11:05.754074 31872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskws89Io/build/release/bin/kudu with pid 24334
I20260810 04:11:05.763429 31872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskws89Io/build/release/bin/kudu with pid 24421
I20260810 04:11:05.769261 31872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskws89Io/build/release/bin/kudu with pid 24507
2026-08-10T04:11:05Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2904 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5787 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (271091 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST